5 papers
A Defense of the Quadratic Model
Alexandru Meterez, Pranav Ajit Nair, Depen Morwani +3
Due to the complexity of neural network loss landscapes, optimization theory is forced to rely on idealized models, and there is generally a tradeoff between how theoretically trac…
Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging
Alexandru Meterez, Pranav Ajit Nair, Depen Morwani +2
Large language models are increasingly trained in continual or open-ended settings, where the total training horizon is not known in advance. Despite this, most existing pretrainin…
Matryoshka Quantization
Pranav Nair, Puranjay Datta, Jeff Dean +2
Quantizing model weights is critical for reducing the communication and inference costs of large models. However, quantizing models -- especially to low precisions like int4 or int…
CDQuant: Greedy Coordinate Descent for Accurate LLM Quantization
Pranav Ajit Nair, Arun Sai Suggala
Large language models (LLMs) have recently demonstrated remarkable performance across diverse language tasks. But their deployment is often constrained by their substantial computa…
Tandem Transformers for Inference Efficient LLMs
Aishwarya P S, Pranav Ajit Nair, Yashas Samaga +4
The autoregressive nature of conventional large language models (LLMs) inherently limits inference speed, as tokens are generated sequentially. While speculative and parallel decod…