978 citations · 1.6k across the 24 of their papers we have counts for
27 papers
Parameter symmetries determine representational geometry in overparameterized nonlinear networks
Marvin Theiss, Lukas Braun, Andrew M. Saxe +1
Representations are routinely used across machine learning, psychology, and neuroscience to draw inferences about the computations of biological and artificial systems. Such infere…
Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks
Lukas Braun, Erin Grant, Andrew M. Saxe
A foundational principle of connectionism is that perception, action, and cognition emerge from parallel computations among simple, interconnected units that generate and rely on n…
On The Specialization of Neural Modules
Devon Jarvis, Richard Klein, Benjamin Rosman +1
A number of machine learning models have been proposed with the goal of achieving systematic generalization: the ability to reason about new situations by combining aspects of prev…
Are task representations gated in macaque prefrontal cortex?
Timo Flesch, Valerio Mante, William Newsome +3
A recent paper (Flesch et al, 2022) describes behavioural and neural data suggesting that task representations are gated in the prefrontal cortex in both humans and macaques. This…
Abrupt and spontaneous strategy switches emerge in simple regularised neural networks
Anika T. Löwe, Léo Touzo, Paul S. Muhle-Karbe +3
Humans sometimes have an insight that leads to a sudden and drastic performance improvement on the task they are working on. Sudden strategy adaptations are often linked to insight…
Continual task learning in natural and artificial agents
Timo Flesch, Andrew Saxe, Christopher Summerfield
How do humans and other animals learn new tasks? A wave of brain recording studies has investigated how neural representations change during task learning, with a focus on how task…
The Neural Race Reduction: Dynamics of Abstraction in Gated Networks
Andrew M. Saxe, Shagun Sodhani, Sam Lewallen
Our theoretical understanding of deep learning has not kept pace with its empirical success. While network architecture is known to be critical, we do not yet understand its effect…
Know your audience: specializing grounded language models with listener subtraction
Aaditya K. Singh, David Ding, Andrew Saxe +2
Effective communication requires adapting to the idiosyncrasies of each communicative context--such as the common ground shared with each partner. Humans demonstrate this ability t…
Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation
Sebastian Lee, Stefano Sarao Mannelli, Claudia Clopath +2
Continual learning - learning new tasks in sequence while maintaining performance on old tasks - remains particularly challenging for artificial neural networks. Surprisingly, the…
Modelling continual learning in humans with Hebbian context gating and exponentially decaying task signals
Timo Flesch, David G. Nagy, Andrew Saxe +1
Humans can learn several tasks in succession with minimal mutual interference but perform more poorly when trained on multiple tasks at once. The opposite is true for standard deep…
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity
Sebastian Lee, Sebastian Goldt, Andrew Saxe
Continual learning-the ability to learn many tasks in sequence-is critical for artificial learning systems. Yet standard training methods for deep networks often suffer from catast…
An Analytical Theory of Curriculum Learning in Teacher-Student Networks
Luca Saglietti, Stefano Sarao Mannelli, Andrew Saxe
In humans and animals, curriculum learning -- presenting data in a curated order - is critical to rapid learning and effective pedagogy. Yet in machine learning, curricula are not…
Probing transfer learning with a model of synthetic correlated datasets
Federica Gerace, Luca Saglietti, Stefano Sarao Mannelli +2
Transfer learning can significantly improve the sample efficiency of neural networks, by exploiting the relatedness between a data-scarce target task and a data-abundant source tas…
If deep learning is the answer, then what is the question?
Andrew Saxe, Stephanie Nelli, Christopher Summerfield
Neuroscience research is undergoing a minor revolution. Recent advances in machine learning and artificial intelligence (AI) research have opened up new ways of thinking about neur…
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup
Sebastian Goldt, Madhu S. Advani, Andrew M. Saxe +2
Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics…
Generalisation dynamics of online learning in over-parameterised neural networks
Sebastian Goldt, Madhu S. Advani, Andrew M. Saxe +2
Deep neural networks achieve stellar generalisation on a variety of problems, despite often being large enough to easily fit all their training data. Here we study the generalisati…
A mathematical theory of semantic development in deep neural networks
Andrew M. Saxe, James L. McClelland, Surya Ganguli
An extensive body of empirical research has revealed remarkable regularities in the acquisition, organization, deployment, and neural representation of human semantic knowledge, th…
Are Efficient Deep Representations Learnable?
Maxwell Nye, Andrew Saxe
Many theories of deep learning have shown that a deep network can require dramatically fewer resources to represent a given function compared to a shallow network. But a question r…
Minnorm training: an algorithm for training over-parameterized deep neural networks
Yamini Bansal, Madhu Advani, David D Cox +1
In this work, we propose a new training method for finding minimum weight norm solutions in over-parameterized neural networks (NNs). This method seeks to improve training speed an…
Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning
Yao Zhang, Andrew M. Saxe, Madhu S. Advani +1
Finding parameters that minimise a loss function is at the core of many machine learning methods. The Stochastic Gradient Descent algorithm is widely used and delivers state of the…
High-dimensional dynamics of generalization error in neural networks
Madhu S. Advani, Andrew M. Saxe
We perform an average case analysis of the generalization dynamics of large neural networks trained using gradient descent. We study the practically-relevant "high-dimensional" reg…
Hierarchical Subtask Discovery With Non-Negative Matrix Factorization
Adam C. Earle, Andrew M. Saxe, Benjamin Rosman
Hierarchical reinforcement learning methods offer a powerful means of planning flexible behavior in complicated domains. However, learning an appropriate hierarchical decomposition…
Hierarchy through Composition with Linearly Solvable Markov Decision Processes
Andrew M. Saxe, Adam Earle, Benjamin Rosman
Hierarchical architectures are critical to the scalability of reinforcement learning methods. Current hierarchical frameworks execute actions serially, with macro-actions comprisin…
Tensor Switching Networks
Chuan-Yung Tsai, Andrew Saxe, David Cox
We present a novel neural network algorithm, the Tensor Switching (TS) network, which generalizes the Rectified Linear Unit (ReLU) nonlinearity to tensor-valued hidden units. The T…