From the 2 of 28 linked papers with an AI index.
28 papers
Efficient Long-Horizon Learning for Learned Optimization
Xiaolong Huang, Benjamin Thérien, James Harrison +1
Learned optimization aims to improve upon hand-designed optimizers (e.g., Adam and Muon) by meta-learning small neural network optimizers over a distribution of tasks. While recent…
Stabilizing Native Low-Rank LLM Pretraining
Paul Janson, Edouard Oyallon, Eugene Belilovsky
The paper presents a method to train large language models from scratch using only low‑rank factorized weights by introducing Spectron, a spectral renormalization and orthogonaliza…
Can Model Merging Improve Aggregation in DiLoCo?
Stefan Horoi, Benjamin Thérien, Guy Wolf +1
Model merging techniques, which aggregate independently finetuned models into one to combine their capabilities, have become a topic of significant interest in recent years, with a…
Stabilizing Black-Box Prompt Optimization with Textual Regularization and Signal Aggregation
MohammadReza Davari, Utkarsh Garg, Weixin Cai +1
An increasing number of NLP applications interact with large language models (LLMs) through black-box APIs, making prompt engineering critical for controlling model behavior. Recen…
From Memorization to Parameter Interference: How Overtraining Experts Harms Model Merging
Stefan Horoi, Guy Wolf, Eugene Belilovsky +1
Modern deep learning is increasingly characterized by the use of open-weight foundation models that can be fine-tuned on specialized datasets. This has led to a proliferation of ex…
Unifying Local Communications and Local Updates for LLM Pretraining
Pietro Cagnasso, Eugene Belilovsky, Edouard Oyallon
Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links. Many pract…