2 citations · 5 across the 5 of their papers we have counts for
9 papers
Learning Representation from Neural Fisher Kernel with Low-rank Approximation
Ruixiang Zhang, Shuangfei Zhai, Etai Littwin +1
In this paper, we study the representation of neural networks from the view of kernels. We first define the Neural Fisher Kernel (NFK), which is the Fisher Kernel applied to neural…
Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks
Shih-Yu Sun, Vimal Thilak, Etai Littwin +2
Deep linear networks trained with gradient descent yield low rank solutions, as is typically studied in matrix factorization. In this paper, we take a step further and analyze impl…
Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
Etai Littwin, Omid Saremi, Shuangfei Zhai +4
We analyze the learning dynamics of infinitely wide neural networks with a finite sized bottle-neck. Unlike the neural tangent kernel limit, a bottleneck in an otherwise infinite w…
Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics
Greg Yang, Etai Littwin
Yang (2020a) recently showed that the Neural Tangent Kernel (NTK) at initialization has an infinite-width limit for a large class of architectures including modern staples such as…
Collegial Ensembles
Etai Littwin, Ben Myara, Sima Sabah +3
Modern neural network performance typically improves as model size increases. A recent line of research on the Neural Tangent Kernel (NTK) of over-parameterized networks indicates…
On Infinite-Width Hypernetworks
Etai Littwin, Tomer Galanti, Lior Wolf +1
{\em Hypernetworks} are architectures that produce the weights of a task-specific {\em primary network}. A notable application of hypernetworks in the recent literature involves le…