204 citations · 504 across the 24 of their papers we have counts for
24 papers
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
Gheorghe Comanici, Eric Bieber, Mike Schaekermann +3431
In this report, we introduce the Gemini 2.X model family: Gemini 2.5 Pro and Gemini 2.5 Flash, as well as our earlier Gemini 2.0 Flash and Flash-Lite models. Gemini 2.5 Pro is our…
Reasoning with Latent Thoughts: On the Power of Looped Transformers
Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li +2
Large language models have shown remarkable reasoning abilities and scaling laws suggest that large parameter count, especially along the depth axis, is the primary driver. In this…
Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding
Tian Jin, Ellie Y. Cheng, Zack Ankner +6
Decoding with autoregressive large language models (LLMs) traditionally occurs sequentially, generating one token after another. An emerging line of work explored parallel decoding…
StagFormer: Time Staggering Transformer Decoding for RunningLayers In Parallel
Dylan Cutler, Arun Kandoor, Nishanth Dikkala +3
Decoding in a Transformer based language model is inherently sequential as a token's embedding needs to pass through all the layers in the network before the generation of the next…
On the Role of Depth and Looping for In-Context Learning with Task Diversity
Khashayar Gatmiry, Nikunj Saunshi, Sashank J. Reddi +2
The intriguing in-context learning (ICL) abilities of deep Transformer models have lately garnered significant attention. By studying in-context linear regression on unimodal Gauss…
A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs
Ankit Singh Rawat, Veeranjaneyulu Sadhanala, Afshin Rostamizadeh +12
A primary challenge in large language model (LLM) development is their onerous pre-training cost. Typically, such pre-training involves optimizing a self-supervised objective (such…
Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?
Khashayar Gatmiry, Nikunj Saunshi, Sashank J. Reddi +2
The remarkable capability of Transformers to do reasoning and few-shot learning, without any fine-tuning, is widely conjectured to stem from their ability to implicitly simulate a…
On the Inductive Bias of Stacking Towards Improving Reasoning
Nikunj Saunshi, Stefani Karp, Shankar Krishnan +3
Given the increasing scale of model sizes, novel training strategies like gradual stacking [Gong et al., 2019, Reddi et al., 2023] have garnered interest. Stacking enables efficien…
Landscape-Aware Growing: The Power of a Little LAG
Stefani Karp, Nikunj Saunshi, Sobhan Miryoosefi +2
Recently, there has been increasing interest in efficient pretraining paradigms for training Transformer-based models. Several recent approaches use smaller models to initialize la…
Efficient Stagewise Pretraining via Progressive Subnetworks
Abhishek Panigrahi, Nikunj Saunshi, Kaifeng Lyu +4
Recent developments in large language models have sparked interest in efficient pretraining methods. Stagewise training approaches to improve efficiency, like gradual stacking and…
Reasoning in Large Language Models Through Symbolic Math Word Problems
Vedant Gaur, Nikunj Saunshi
Large language models (LLMs) have revolutionized NLP by solving downstream tasks with little to no labeled data. Despite their versatile abilities, the larger question of their abi…
Task-Specific Skill Localization in Fine-tuned Language Models
Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao +1
Pre-trained language models can be fine-tuned to solve diverse NLP tasks, including in few-shot settings. Thus fine-tuning allows the model to quickly pick up task-specific ``skill…
New Definitions and Evaluations for Saliency Methods: Staying Intrinsic, Complete and Sound
Arushi Gupta, Nikunj Saunshi, Dingli Yu +2
Saliency methods compute heat maps that highlight portions of an input that were most {\em important} for the label assigned to it by a deep net. Evaluations of saliency methods co…
Understanding Influence Functions and Datamodels via Harmonic Analysis
Nikunj Saunshi, Arushi Gupta, Mark Braverman +1
Influence functions estimate effect of individual data points on predictions of the model on test data and were adapted to deep learning in Koh and Liang [2017]. They have been use…
Understanding Contrastive Learning Requires Incorporating Inductive Biases
Nikunj Saunshi, Jordan Ash, Surbhi Goel +5
Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmenta…
On Predicting Generalization using GANs
Yi Zhang, Arushi Gupta, Nikunj Saunshi +1
Research on generalization bounds for deep networks seeks to give ways to predict test error using just the training dataset and the network parameters. While generalization bounds…
A Representation Learning Perspective on the Importance of Train-Validation Splitting in Meta-Learning
Nikunj Saunshi, Arushi Gupta, Wei Hu
An effective approach in meta-learning is to utilize multiple "train tasks" to learn a good initialization for model parameters that can help solve unseen "test tasks" with very fe…
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Nikunj Saunshi, Sadhika Malladi, Sanjeev Arora
Autoregressive language models, pretrained using large text corpora to do well on next word prediction, have been successful at solving many downstream tasks, even with zero-shot u…
Predicting What You Already Know Helps: Provable Self-Supervised Learning
Jason D. Lee, Qi Lei, Nikunj Saunshi +1
Self-supervised representation learning solves auxiliary prediction tasks (known as pretext tasks) without requiring labeled data to learn useful semantic representations. These pr…
A Sample Complexity Separation between Non-Convex and Convex Meta-Learning
Nikunj Saunshi, Yi Zhang, Mikhail Khodak +1
One popular trend in meta-learning is to learn from many training tasks a common initialization for a gradient-based method that can be used to solve a new task with few samples. T…
Provable Representation Learning for Imitation Learning via Bi-level Optimization
Sanjeev Arora, Simon S. Du, Sham Kakade +2
A common strategy in modern learning systems is to learn a representation that is useful for many tasks, a.k.a. representation learning. We study this strategy in the imitation lea…
A Theoretical Analysis of Contrastive Unsupervised Representation Learning
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak +2
Recent empirical works have successfully used unlabeled data to learn feature representations that are broadly useful in downstream classification tasks. Several of these methods a…
A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors
Mikhail Khodak, Nikunj Saunshi, Yingyu Liang +3
Motivations like domain adaptation, transfer learning, and feature learning have fueled interest in inducing embeddings for rare or unseen words, n-grams, synsets, and other textua…
A Large Self-Annotated Corpus for Sarcasm
Mikhail Khodak, Nikunj Saunshi, Kiran Vodrahalli
We introduce the Self-Annotated Reddit Corpus (SARC), a large corpus for sarcasm research and for training and evaluating systems for sarcasm detection. The corpus has 1.3 million…