4 papers
A Defense of the Quadratic Model
Alexandru Meterez, Pranav Ajit Nair, Depen Morwani +3
Due to the complexity of neural network loss landscapes, optimization theory is forced to rely on idealized models, and there is generally a tradeoff between how theoretically trac…
Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging
Alexandru Meterez, Pranav Ajit Nair, Depen Morwani +2
Large language models are increasingly trained in continual or open-ended settings, where the total training horizon is not known in advance. Despite this, most existing pretrainin…
Matryoshka Quantization
Pranav Nair, Puranjay Datta, Jeff Dean +2
Quantizing model weights is critical for reducing the communication and inference costs of large models. However, quantizing models -- especially to low precisions like int4 or int…
CDQuant: Greedy Coordinate Descent for Accurate LLM Quantization
Pranav Ajit Nair, Arun Sai Suggala
Large language models (LLMs) have recently demonstrated remarkable performance across diverse language tasks. But their deployment is often constrained by their substantial computa…