130 citations · 272 across the 11 of their papers we have counts for
13 papers
FlashOptim: Optimizers for Memory-Efficient Training
Jose Javier Gonzalez Ortiz, Abhay Gupta, Christopher Rinard +1
Standard mixed-precision training of neural networks requires many bytes of accelerator memory for each model parameter. These bytes reflect not just the parameter itself, but also…
nit Scaling: Simple and Scalable FP8 LLM Training
Saaketh Narayan, Abhay Gupta, Mansheej Paul +1
Large Language Model training with 8-bit floating point (FP8) formats promises significant efficiency improvements, but reduced numerical precision makes training challenging. It i…
Dynamic Masking Rate Schedules for MLM Pretraining
Zachary Ankner, Naomi Saphra, Davis Blalock +2
Most works on transformers trained with the Masked Language Modeling (MLM) objective use the original BERT model's fixed masking rate of 15%. We propose to instead dynamically sche…
Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities
Brian R. Bartoldson, Bhavya Kailkhura, Davis Blalock
Although deep learning has made great progress in recent years, the exploding economic and environmental costs of training neural networks are becoming unsustainable. To address th…
Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates
Jacob Portes, Davis Blalock, Cory Stephenson +1
Benchmarking the tradeoff between neural network accuracy and training time is computationally expensive. Here we show how a multiplicative cyclic learning rate schedule can be use…
Multiplying Matrices Without Multiplying
Davis Blalock, John Guttag
Multiplying matrices is among the most fundamental and compute-intensive operations in machine learning. Consequently, there has been significant work on efficiently approximating…