111 citations · 403 across the 22 of their papers we have counts for
23 papers
Deep sequence models tend to memorize geometrically; it is unclear why
Shahriar Noroozizadeh, Vaishnavh Nagarajan, Elan Rosenfeld +1
Deep sequence models are said to store atomic facts predominantly in the form of associative memory: a brute-force lookup of co-occurring entities. We identify a dramatically diffe…
Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production
Alexandre Galashov, Matt Jones, Rosemary Ke +3
Within the landscape of inference-time scaling methods for foundation models, a width-based approach to scaling -- which involves the insertion of <pause> tokens in the input strea…
Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction
Vaishnavh Nagarajan, Chen Henry Wu, Charles Ding +1
We design a suite of minimal algorithmic tasks that are a loose abstraction of open-ended real-world tasks. This allows us to cleanly and controllably quantify the creative limits…
Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning
Jacob Mitchell Springer, Vaishnavh Nagarajan, Aditi Raghunathan
Sharpness-Aware Minimization (SAM) has emerged as a promising alternative optimizer to stochastic gradient descent (SGD). The originally-proposed motivation behind SAM was to bias…
The pitfalls of next-token prediction
Gregor Bachmann, Vaishnavh Nagarajan
Can a mere next-token predictor faithfully model human intelligence? We crystallize this emerging concern and correct popular misconceptions surrounding it, and advocate a simple m…
What do larger image classifiers memorise?
Michal Lukasik, Vaishnavh Nagarajan, Ankit Singh Rawat +2
The success of modern neural networks has prompted study of the connection between memorisation and generalisation: overparameterised models generalise well, despite being able to…
The Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learning
Tian Jin, Nolan Clement, Xin Dong +4
How does scaling the number of parameters in large language models (LLMs) affect their core capabilities? We study two natural scaling techniques -- weight pruning and simply train…
Think before you speak: Training Language Models With Pause Tokens
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat +3
Language models generate responses by producing a series of tokens in immediate succession: the token is an outcome of manipulating hidden vectors per layer, one v…
ResMem: Learn what you can and memorize the rest
Zitong Yang, Michal Lukasik, Vaishnavh Nagarajan +5
The impressive generalization performance of modern neural networks is attributed in part to their ability to implicitly memorize complex training patterns. Inspired by this, we ex…
On student-teacher deviations in distillation: does it pay to disobey?
Vaishnavh Nagarajan, Aditya Krishna Menon, Srinadh Bhojanapalli +2
Knowledge distillation (KD) has been widely used to improve the test accuracy of a "student" network, by training it to mimic the soft probabilities of a trained "teacher" network.…
Explaining generalization in deep learning: progress and fundamental limits
Vaishnavh Nagarajan
This dissertation studies a fundamental open challenge in deep learning theory: why do deep networks generalize well even while being overparameterized, unregularized and fitting t…
Assessing Generalization of SGD via Disagreement
Yiding Jiang, Vaishnavh Nagarajan, Christina Baek +1
We empirically show that the test error of deep networks can be estimated by simply training the same architecture on the same training set but with a different run of Stochastic G…
A Learning Theoretic Perspective on Local Explainability
Jeffrey Li, Vaishnavh Nagarajan, Gregory Plumb +1
In this paper, we explore connections between interpretable machine learning and learning theory through the lens of local approximation explanations. First, we tackle the traditio…
Understanding the Failure Modes of Out-of-Distribution Generalization
Vaishnavh Nagarajan, Anders Andreassen, Behnam Neyshabur
Empirical studies suggest that machine learning models often rely on features, such as the background, that may be spuriously correlated with the label only during training time, r…
Provably Safe PAC-MDP Exploration Using Analogies
Melrose Roderick, Vaishnavh Nagarajan, J. Zico Kolter
A key challenge in applying reinforcement learning to safety-critical domains is understanding how to balance exploration (needed to attain good performance on the task) with safet…
Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience
Vaishnavh Nagarajan, J. Zico Kolter
The ability of overparameterized deep networks to generalize well has been linked to the fact that stochastic gradient descent (SGD) finds solutions that lie in flat, wide minima i…
Uniform convergence may be unable to explain generalization in deep learning
Vaishnavh Nagarajan, J. Zico Kolter
Aimed at explaining the surprisingly good generalization behavior of overparameterized deep networks, recent works have developed a variety of generalization bounds for deep learni…
Generalization in Deep Networks: The Role of Distance from Initialization
Vaishnavh Nagarajan, J. Zico Kolter
Why does training deep neural networks using stochastic gradient descent (SGD) result in a generalization error that does not worsen with the number of parameters in the network? T…
Revisiting Adversarial Risk
Arun Sai Suggala, Adarsh Prasad, Vaishnavh Nagarajan +1
Recent works on adversarial perturbations show that there is an inherent trade-off between standard test accuracy and adversarial accuracy. Specifically, they show that no classifi…
Lifelong Learning in Costly Feature Spaces
Maria-Florina Balcan, Avrim Blum, Vaishnavh Nagarajan
An important long-term goal in machine learning systems is to build learning agents that, like humans, can learn many tasks over their lifetime, and moreover use information from t…
Gradient descent GAN optimization is locally stable
Vaishnavh Nagarajan, J. Zico Kolter
Despite the growing prominence of generative adversarial networks (GANs), optimization in GANs is still a poorly understood topic. In this paper, we analyze the "gradient descent"…
Learning-Theoretic Foundations of Algorithm Configuration for Combinatorial Partitioning Problems
Maria-Florina Balcan, Vaishnavh Nagarajan, Ellen Vitercik +1
Max-cut, clustering, and many other partitioning problems that are of significant importance to machine learning and other scientific fields are NP-hard, a reality that has motivat…
A Reinforcement Learning Approach to Online Learning of Decision Trees
Abhinav Garlapati, Aditi Raghunathan, Vaishnavh Nagarajan +1
Online decision tree learning algorithms typically examine all features of a new data point to update model parameters. We propose a novel alternative, Reinforcement Learning- base…