12 citations · 50 across the 49 of their papers we have counts for
11 papers · 2 filters
Test Time Adaptation Using Adaptive Quantile Recalibration
Paria Mehrbod, Pedro Vianna, Geraldin Nanfack +2
Domain adaptation is a key strategy for enhancing the generalizability of deep learning models in real-world scenarios, where test distributions often diverge significantly from th…
Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clients
Gwen Legate, Irina Rish, Eugene Belilovsky
Federated learning enables collaborative model training across numerous edge devices without requiring participants to share data; however, memory and communication constraints on…
Overcoming the Communication-Performance Tradeoff in LLM Pretraining
Amir Sarfi, Benjamin Thérien, Joel Lidin +1
Communication-efficient distributed training algorithms (e.g., DiLoCo) have received considerable interest due to their benefits for training large language models (LLMs) in bandwi…
Stabilizing Black-Box Prompt Optimization with Textual Regularization and Signal Aggregation
MohammadReza Davari, Utkarsh Garg, Weixin Cai +1
An increasing number of NLP applications interact with large language models (LLMs) through black-box APIs, making prompt engineering critical for controlling model behavior. Recen…
Model Parallelism With Subnetwork Data Parallelism
Vaibhav Singh, Zafir Khalid, Pietro Cagnasso +2
Pre-training large neural networks at scale imposes heavy memory demands on accelerators and often requires costly communication. We introduce Subnetwork Data Parallelism (SDP), a…
From Memorization to Parameter Interference: How Overtraining Experts Harms Model Merging
Stefan Horoi, Guy Wolf, Eugene Belilovsky +1
Modern deep learning is increasingly characterized by the use of open-weight foundation models that can be fine-tuned on specialized datasets. This has led to a proliferation of ex…
PyLO: Towards Accessible Learned Optimizers in PyTorch
Paul Janson, Benjamin Therien, Quentin Anthony +3
Learned optimizers have been an active research topic over the past decade, with increasing progress toward practical, general-purpose optimizers that can serve as drop-in replacem…
MuLoCo: Muon is a practical inner optimizer for DiLoCo
Benjamin Thérien, Xiaolong Huang, Aaron Defazio +2
DiLoCo is a powerful framework for training large language models (LLMs), enabling larger optimal batch sizes and increased accelerator utilization under networking constraints. Ho…
Continual Pre-training of MoEs: How robust is your router?
Benjamin Thérien, Charles-Étienne Joseph, Zain Sarwar +7
Sparsely-activated Mixture of Experts (MoE) transformers are promising architectures for foundation models. Compared to dense transformers that require the same amount of floating-…
FairDropout: Using Example-Tied Dropout to Enhance Generalization of Minority Groups
Geraldin Nanfack, Eugene Belilovsky
Deep learning models frequently exploit spurious features in training data to achieve low training error, often resulting in poor generalization when faced with shifted testing dis…
Celo: Training Versatile Learned Optimizers on a Compute Diet
Abhinav Moudgil, Boris Knyazev, Guillaume Lajoie +1
Learned optimization has emerged as a promising alternative to hand-crafted optimizers, with the potential to discover stronger learned update rules that enable faster, hyperparame…