14 citations · 14 across the 9 of their papers we have counts for
10 papers · 1 filter
Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerf
Priya Kasimbeg, Ahmed Khaled, Aahlad Chandrabhatta +9
Evaluating deep learning optimizer is challenging because confounding factors such as hyperparameter tuning and optimizer-model co-design often obscure true algorithmic performance…
Amortizing Scaling Law Construction Costs
Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6
Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and paramete…
Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss
Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4
We study the scaling behavior of learning rate and batch size in pretraining dense large language models on English-prevalent corpora. Beyond scaling jointly optimal learning rates…
Deriving Hyperparameter Scaling Laws via Modern Optimization Theory
Egor Shulgin, Dimitri von Rütte, Tianyue H. Zhang +3
Hyperparameter transfer has become an important component of modern large-scale training recipes. Existing methods, such as muP, primarily focus on transfer between model sizes, wi…
Training Dynamics Impact Post-Training Quantization Robustness
Albert Catalan-Tatjer, Niccolò Ajroldi, Jonas Geiping
While post-training quantization is widely adopted for efficient deployment of large language models, the mechanisms underlying quantization robustness remain unclear. We conduct a…
Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison
Marianna Nezhurina, Harsh Raj, Jörg Franke +7
Reliable progress in language modeling depends on knowing whether a new dataset, architecture, or training recipe is genuinely better than existing alternatives. To foster reproduc…
Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size
Rustem Islamov, Niccolo Ajroldi, Antonio Orvieto +1
Modern optimization algorithms that incorporate momentum and adaptive step-size offer improved performance in numerous challenging deep learning tasks. However, their effectiveness…
Can you Finetune your Binoculars? Embedding Text Watermarks into the Weights of Large Language Models
Fay Elhassan, Niccolò Ajroldi, Antonio Orvieto +1
The indistinguishability of AI-generated content from human text raises challenges in transparency and accountability. While several methods exist to watermark models behind APIs,…
When, Where and Why to Average Weights?
Niccolò Ajroldi, Antonio Orvieto, Jonas Geiping
Averaging checkpoints along the training trajectory is a simple yet powerful approach to improve the generalization performance of Machine Learning models and reduce training time.…
Loss Landscape Characterization of Neural Networks without Over-Parametrization
Rustem Islamov, Niccolò Ajroldi, Antonio Orvieto +1
Optimization methods play a crucial role in modern machine learning, powering the remarkable empirical achievements of deep learning models. These successes are even more remarkabl…