Publication

On-line learning dynamics of ReLU neural networks using statistical physics techniques

Straat, M. & Biehl, M., 18-Mar-2019, In : ArXiv e-prints. 1903.07378, 1903.07378

Research output: Contribution to journalArticleAcademicpeer-review

We introduce exact macroscopic on-line learning dynamics of two-layer neural networks with ReLU units in the form of a system of differential equations, using techniques borrowed from statistical physics. For the first experiments, numerical solutions reveal similar behavior compared to sigmoidal activation researched in earlier work. In these experiments the theoretical results show good correspondence with simulations. In ove-rrealizable and unrealizable learning scenarios, the learning behavior of ReLU networks shows distinctive characteristics compared to sigmoidal networks.
Original languageEnglish
JournalArXiv e-prints
Volume1903.07378
Issue number1903.07378
Publication statusPublished - 18-Mar-2019

    Keywords

  • cs.LG, cond-mat.dis-nn, stat.ML
Related Activities
  1. European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning

    Michiel Straat (Speaker), Michael Biehl (Participant)
    24-Apr-201926-Apr-2019

    Activity: Participating in or organising an eventParticipation in conferenceAcademic

View all (1) »

View graph of relations

Download statistics

No data available

ID: 78117626