567 citations · 1k across the 10 of their papers we have counts for
4 papers · 1 filter
A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs
Ankit Singh Rawat, Veeranjaneyulu Sadhanala, Afshin Rostamizadeh +12
A primary challenge in large language model (LLM) development is their onerous pre-training cost. Typically, such pre-training involves optimizing a self-supervised objective (such…
Nonlinear Initialization Methods for Low-Rank Neural Networks
Kiran Vodrahalli, Rakesh Shivanna, Maheswaran Sathiamoorthy +2
We propose a novel low-rank initialization framework for training low-rank deep neural networks -- networks where the weight parameters are re-parameterized by products of two low-…
Understanding and Improving Knowledge Distillation
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao +4
Knowledge Distillation (KD) is a model-agnostic technique to improve model quality while having a fixed capacity budget. It is a commonly used technique for model compression, wher…
How Many Pairwise Preferences Do We Need to Rank A Graph Consistently?
Aadirupa Saha, Rakesh Shivanna, Chiranjib Bhattacharyya
We consider the problem of optimal recovery of true ranking of items from a randomly chosen subset of their pairwise preferences. It is well known that without any further assu…