2.3k citations · 10.3k across the 79 of their papers we have counts for
8 papers · 2 filters
On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers
Weinan E, Stephan Wojtowytsch
A recent numerical study observed that neural network classifiers enjoy a large degree of symmetry in the penultimate layer. Namely, if where is a linear map…
Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
Pan Zhou, Jiashi Feng, Chao Ma +3
It is not clear yet why ADAM-alike adaptive gradient algorithms suffer from worse generalization performance than SGD despite their faster training speed. This work aims to provide…
Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
Weinan E, Chao Ma, Stephan Wojtowytsch +1
The purpose of this article is to review the achievements made in the last few years towards the understanding of the reasons behind the success and subtleties of neural network-ba…
On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis
Zhong Li, Jiequn Han, Weinan E +1
We study the approximation properties and optimization dynamics of recurrent neural networks (RNNs) when applied to learn input-output relationships in temporal data. We consider t…
A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms
Chao Ma, Lei Wu, Weinan E
The dynamic behavior of RMSprop and Adam algorithms is studied through a combination of careful numerical experiments and theoretical explanations. Three types of qualitative featu…
The Slow Deterioration of the Generalization Error of the Random Feature Model
Chao Ma, Lei Wu, Weinan E
The random feature model exhibits a kind of resonance behavior when the number of parameters is close to the training sample size. This behavior is characterized by the appearance…