8 citations · 42 across the 24 of their papers we have counts for
Showing 2021Show all
3 papers · 1 filter
cs.LG2021
Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks
Shih-Yu Sun, Vimal Thilak, Etai Littwin +2
Deep linear networks trained with gradient descent yield low rank solutions, as is typically studied in matrix factorization. In this paper, we take a step further and analyze impl…
cs.LG2021
Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
Etai Littwin, Omid Saremi, Shuangfei Zhai +4
We analyze the learning dynamics of infinitely wide neural networks with a finite sized bottle-neck. Unlike the neural tangent kernel limit, a bottleneck in an otherwise infinite w…
cs.LG2021
Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics
Greg Yang, Etai Littwin
Yang (2020a) recently showed that the Neural Tangent Kernel (NTK) at initialization has an infinite-width limit for a large class of architectures including modern staples such as…