1 citations · 1 across the 5 of their papers we have counts for
4 papers · 1 filter
LOTION: Smoothing the Optimization Landscape for Quantized Training
Mujin Kwun, Depen Morwani, Chloe Huangyuan Su +3
Optimizing neural networks for quantized objectives is fundamentally challenging because the quantizer is piece-wise constant, yielding zero gradients everywhere except at quantiza…
Characterization and Mitigation of Training Instabilities in Microscaling Formats
Huangyuan Su, Mujin Kwun, Stephanie Gil +2
Training large language models is an expensive, compute-bound process that must be repeated as models scale, algorithms improve, and new data is collected. To address this, next-ge…
Loss-to-Loss Prediction: Scaling Laws for All Datasets
David Brandfonbrener, Nikhil Anand, Nikhil Vyas +2
While scaling laws provide a reliable methodology for predicting train loss across compute scales for a single data distribution, less is known about how these predictions should c…
Mixture of Parrots: Experts improve memorization more than reasoning
Samy Jelassi, Clara Mohri, David Brandfonbrener +7
The Mixture-of-Experts (MoE) architecture enables a significant increase in the total number of model parameters with minimal computational overhead. However, it is not clear what…