978 citations · 1.6k across the 24 of their papers we have counts for
11 papers · 1 filter
Parameter symmetries determine representational geometry in overparameterized nonlinear networks
Marvin Theiss, Lukas Braun, Andrew M. Saxe +1
Representations are routinely used across machine learning, psychology, and neuroscience to draw inferences about the computations of biological and artificial systems. Such infere…
Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks
Lukas Braun, Erin Grant, Andrew M. Saxe
A foundational principle of connectionism is that perception, action, and cognition emerge from parallel computations among simple, interconnected units that generate and rely on n…
On The Specialization of Neural Modules
Devon Jarvis, Richard Klein, Benjamin Rosman +1
A number of machine learning models have been proposed with the goal of achieving systematic generalization: the ability to reason about new situations by combining aspects of prev…
The Neural Race Reduction: Dynamics of Abstraction in Gated Networks
Andrew M. Saxe, Shagun Sodhani, Sam Lewallen
Our theoretical understanding of deep learning has not kept pace with its empirical success. While network architecture is known to be critical, we do not yet understand its effect…
Know your audience: specializing grounded language models with listener subtraction
Aaditya K. Singh, David Ding, Andrew Saxe +2
Effective communication requires adapting to the idiosyncrasies of each communicative context--such as the common ground shared with each partner. Humans demonstrate this ability t…
An Analytical Theory of Curriculum Learning in Teacher-Student Networks
Luca Saglietti, Stefano Sarao Mannelli, Andrew Saxe
In humans and animals, curriculum learning -- presenting data in a curated order - is critical to rapid learning and effective pedagogy. Yet in machine learning, curricula are not…
Probing transfer learning with a model of synthetic correlated datasets
Federica Gerace, Luca Saglietti, Stefano Sarao Mannelli +2
Transfer learning can significantly improve the sample efficiency of neural networks, by exploiting the relatedness between a data-scarce target task and a data-abundant source tas…
A mathematical theory of semantic development in deep neural networks
Andrew M. Saxe, James L. McClelland, Surya Ganguli
An extensive body of empirical research has revealed remarkable regularities in the acquisition, organization, deployment, and neural representation of human semantic knowledge, th…
Are Efficient Deep Representations Learnable?
Maxwell Nye, Andrew Saxe
Many theories of deep learning have shown that a deep network can require dramatically fewer resources to represent a given function compared to a shallow network. But a question r…
Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning
Yao Zhang, Andrew M. Saxe, Madhu S. Advani +1
Finding parameters that minimise a loss function is at the core of many machine learning methods. The Stochastic Gradient Descent algorithm is widely used and delivers state of the…
Active Long Term Memory Networks
Tommaso Furlanello, Jiaping Zhao, Andrew M. Saxe +2
Continual Learning in artificial neural networks suffers from interference and forgetting when different tasks are learned sequentially. This paper introduces the Active Long Term…