From the 2 of 28 linked papers with an AI index.
11 papers · 1 filter
Continual Pre-training of MoEs: How robust is your router?
Benjamin Thérien, Charles-Ãtienne Joseph, Zain Sarwar +7
Sparsely-activated Mixture of Experts (MoE) transformers are promising architectures for foundation models. Compared to dense transformers that require the same amount of floating-…
Test Time Adaptation Using Adaptive Quantile Recalibration
Paria Mehrbod, Pedro Vianna, Geraldin Nanfack +2
Domain adaptation is a key strategy for enhancing the generalizability of deep learning models in real-world scenarios, where test distributions often diverge significantly from th…
Overcoming the Communication-Performance Tradeoff in LLM Pretraining
Amir Sarfi, Benjamin Thérien, Joel Lidin +1
Communication-efficient distributed training algorithms (e.g., DiLoCo) have received considerable interest due to their benefits for training large language models (LLMs) in bandwi…
ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
Adel Nabli, Louis Fournier, Pierre Erbacher +3
Training LLMs relies on distributed implementations using multiple GPUs to compute gradients in parallel with sharded optimizers. However, synchronizing gradients in data parallel…
Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clients
Gwen Legate, Irina Rish, Eugene Belilovsky
Federated learning enables collaborative model training across numerous edge devices without requiring participants to share data; however, memory and communication constraints on…
Celo: Training Versatile Learned Optimizers on a Compute Diet
Abhinav Moudgil, Boris Knyazev, Guillaume Lajoie +1
Learned optimization has emerged as a promising alternative to hand-crafted optimizers, with the potential to discover stronger learned update rules that enable faster, hyperparame…