23 citations · 52 across the 21 of their papers we have counts for
21 papers
Do Thinking Tokens Help with Safety?
Narutatsu Ri, Abhishek Panigrahi, Sanjeev Arora
Today's reasoning models use thinking tokens to attain stronger performance on benchmarks than their instruction-tuned counterparts. It is also generally believed that this more "d…
Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
Yinghui He, Simran Kaur, Adithya Bhaskar +7
Current post-training methods in verifiable settings fall into two categories. Reinforcement learning (RLVR) relies on binary rewards, which are broadly applicable and powerful, bu…
In Good GRACEs: Principled Teacher Selection for Knowledge Distillation
Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi +2
Knowledge distillation is an efficient strategy to use data generated by large "teacher" language models to train smaller capable "student" models, but selecting the optimal teache…
Skill-Targeted Adaptive Training
Yinghui He, Abhishek Panigrahi, Yong Lin +1
Language models often show little to no improvement (i.e., "saturation") when trained via vanilla supervised fine-tuning (SFT) on data similar to what they saw in their training se…
AdaptMI: Adaptive Skill-based In-context Math Instruction for Small Language Models
Yinghui He, Abhishek Panigrahi, Yong Lin +1
In-context learning (ICL) allows a language model to improve its problem-solving capability when provided with suitable information in context. Since the choice of in-context infor…
On the Power of Context-Enhanced Learning in LLMs
Xingyu Zhu, Abhishek Panigrahi, Sanjeev Arora
We formalize a new concept for LLMs, context-enhanced learning. It involves standard gradient-based learning on text except that the context is enhanced with additional data on whi…
Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Simon Park, Abhishek Panigrahi, Yun Cheng +3
Vision Language Models (VLMs) are impressive at visual question answering and image captioning. But they underperform on multi-step visual reasoning -- even compared to LLMs on the…
Progressive distillation induces an implicit curriculum
Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi +2
Knowledge distillation leverages a teacher model to improve the training of a student model. A persistent challenge is that a better teacher does not always yield a better student,…
Representing Rule-based Chatbots with Transformers
Dan Friedman, Abhishek Panigrahi, Danqi Chen
What kind of internal mechanisms might Transformers use to conduct fluid, natural-sounding conversations? Prior work has illustrated by construction how Transformers can solve vari…
Efficient Stagewise Pretraining via Progressive Subnetworks
Abhishek Panigrahi, Nikunj Saunshi, Kaifeng Lyu +4
Recent developments in large language models have sparked interest in efficient pretraining methods. Stagewise training approaches to improve efficiency, like gradual stacking and…
Trainable Transformer in Transformer
Abhishek Panigrahi, Sadhika Malladi, Mengzhou Xia +1
Recent works attribute the capability of in-context learning (ICL) in large pre-trained language models to implicitly simulating and fine-tuning an internal model (e.g., linear or…
Do Transformers Parse while Predicting the Masked Word?
Haoyu Zhao, Abhishek Panigrahi, Rong Ge +1
Pre-trained language models have been shown to encode linguistic structures, e.g. dependency and constituency parse trees, in their embeddings while being trained on unsupervised l…
Task-Specific Skill Localization in Fine-tuned Language Models
Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao +1
Pre-trained language models can be fine-tuned to solve diverse NLP tasks, including in few-shot settings. Thus fine-tuning allows the model to quickly pick up task-specific ``skill…
On the SDEs and Scaling Rules for Adaptive Gradient Algorithms
Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi +1
Approximating Stochastic Gradient Descent (SGD) as a Stochastic Differential Equation (SDE) has allowed researchers to enjoy the benefits of studying a continuous optimization traj…
Understanding Gradient Descent on Edge of Stability in Deep Learning
Sanjeev Arora, Zhiyuan Li, Abhishek Panigrahi
Deep learning experiments by Cohen et al. [2021] using deterministic Gradient Descent (GD) revealed an Edge of Stability (EoS) phase when learning rate (LR) and sharpness (i.e., th…
Learning and Generalization in RNNs
Abhishek Panigrahi, Navin Goyal
Simple recurrent neural networks (RNNs) and their more advanced cousins LSTMs etc. have been very successful in sequence modeling. Their theoretical understanding, however, is lack…
Non-Gaussianity of Stochastic Gradient Noise
Abhishek Panigrahi, Raghav Somani, Navin Goyal +1
What enables Stochastic Gradient Descent (SGD) to achieve better generalization than Gradient Descent (GD) in Neural Network training? This question has attracted much attention. I…
Effect of Activation Functions on the Training of Overparametrized Neural Nets
Abhishek Panigrahi, Abhishek Shetty, Navin Goyal
It is well-known that overparametrized neural networks trained using gradient-based methods quickly achieve small training error with appropriate hyperparameter settings. Recent pa…
DeepTagRec: A Content-cum-User based Tag Recommendation Framework for Stack Overflow
Suman Kalyan Maity, Abhishek Panigrahi, Sayan Ghosh +3
In this paper, we develop a content-cum-user based deep learning framework DeepTagRec to recommend appropriate question tags on Stack Overflow. The proposed system learns the conte…
Analysis on Gradient Propagation in Batch Normalized Residual Networks
Abhishek Panigrahi, Yueru Chen, C. -C. Jay Kuo
We conduct mathematical analysis on the effect of batch normalization (BN) on gradient backpropogation in residual network training, which is believed to play a critical role in ad…
Analyzing Social Book Reading Behavior on Goodreads and how it predicts Amazon Best Sellers
Suman Kalyan Maity, Abhishek Panigrahi, Animesh Mukherjee
A book's success/popularity depends on various parameters - extrinsic and intrinsic. In this paper, we study how the book reading characteristics might influence the popularity of…