Publication

On-line learning dynamics of ReLU neural networks using statistical physics techniques

Straat, M. & Biehl, M., 18-Mar-2019, In : ArXiv e-prints. 1903.07378, 1903.07378

Research output: Contribution to journalArticleAcademicpeer-review

APA

Straat, M., & Biehl, M. (2019). On-line learning dynamics of ReLU neural networks using statistical physics techniques. ArXiv e-prints, 1903.07378(1903.07378).

Author

Straat, Michiel ; Biehl, Michael. / On-line learning dynamics of ReLU neural networks using statistical physics techniques. In: ArXiv e-prints. 2019 ; Vol. 1903.07378, No. 1903.07378.

Harvard

Straat, M & Biehl, M 2019, 'On-line learning dynamics of ReLU neural networks using statistical physics techniques', ArXiv e-prints, vol. 1903.07378, no. 1903.07378.

Standard

On-line learning dynamics of ReLU neural networks using statistical physics techniques. / Straat, Michiel; Biehl, Michael.

In: ArXiv e-prints, Vol. 1903.07378, No. 1903.07378, 18.03.2019.

Research output: Contribution to journalArticleAcademicpeer-review

Vancouver

Straat M, Biehl M. On-line learning dynamics of ReLU neural networks using statistical physics techniques. ArXiv e-prints. 2019 Mar 18;1903.07378(1903.07378).


BibTeX

@article{92f41e47542a4c2ca67f65d6fcd85890,
title = "On-line learning dynamics of ReLU neural networks using statistical physics techniques",
abstract = "We introduce exact macroscopic on-line learning dynamics of two-layer neural networks with ReLU units in the form of a system of differential equations, using techniques borrowed from statistical physics. For the first experiments, numerical solutions reveal similar behavior compared to sigmoidal activation researched in earlier work. In these experiments the theoretical results show good correspondence with simulations. In ove-rrealizable and unrealizable learning scenarios, the learning behavior of ReLU networks shows distinctive characteristics compared to sigmoidal networks.",
keywords = "cs.LG, cond-mat.dis-nn, stat.ML",
author = "Michiel Straat and Michael Biehl",
note = "Accepted contribution: ESANN 2019, 6 pages European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning 2019",
year = "2019",
month = "3",
day = "18",
language = "English",
volume = "1903.07378",
journal = "ArXiv e-prints",
number = "1903.07378",

}

RIS

TY - JOUR

T1 - On-line learning dynamics of ReLU neural networks using statistical physics techniques

AU - Straat, Michiel

AU - Biehl, Michael

N1 - Accepted contribution: ESANN 2019, 6 pages European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning 2019

PY - 2019/3/18

Y1 - 2019/3/18

N2 - We introduce exact macroscopic on-line learning dynamics of two-layer neural networks with ReLU units in the form of a system of differential equations, using techniques borrowed from statistical physics. For the first experiments, numerical solutions reveal similar behavior compared to sigmoidal activation researched in earlier work. In these experiments the theoretical results show good correspondence with simulations. In ove-rrealizable and unrealizable learning scenarios, the learning behavior of ReLU networks shows distinctive characteristics compared to sigmoidal networks.

AB - We introduce exact macroscopic on-line learning dynamics of two-layer neural networks with ReLU units in the form of a system of differential equations, using techniques borrowed from statistical physics. For the first experiments, numerical solutions reveal similar behavior compared to sigmoidal activation researched in earlier work. In these experiments the theoretical results show good correspondence with simulations. In ove-rrealizable and unrealizable learning scenarios, the learning behavior of ReLU networks shows distinctive characteristics compared to sigmoidal networks.

KW - cs.LG

KW - cond-mat.dis-nn

KW - stat.ML

M3 - Article

VL - 1903.07378

JO - ArXiv e-prints

JF - ArXiv e-prints

IS - 1903.07378

ER -

ID: 78117626