5 papers
How Inner Optimizers Shape Replica Scaling
Eugene Belilovsky, Stefan Horoi, Edouard Oyallon +1
Training large language models (LLMs) across geographically distributed and poorly interconnected compute pools or data centers has become a topic of significant interest. This has…
Can Model Merging Improve Aggregation in DiLoCo?
Stefan Horoi, Benjamin Thérien, Guy Wolf +1
Model merging techniques, which aggregate independently finetuned models into one to combine their capabilities, have become a topic of significant interest in recent years, with a…
Leveraging Parameter Space Symmetries for Reasoning Skill Transfer in LLMs
Stefan Horoi, Sangwoo Cho, Supriyo Chakraborty +4
Task arithmetic is a powerful technique for transferring skills between Large Language Models (LLMs), but it often suffers from negative interference when models have diverged duri…
From Memorization to Parameter Interference: How Overtraining Experts Harms Model Merging
Stefan Horoi, Guy Wolf, Eugene Belilovsky +1
Modern deep learning is increasingly characterized by the use of open-weight foundation models that can be fine-tuned on specialized datasets. This has led to a proliferation of ex…
Non-Uniform Parameter-Wise Model Merging
Albert Manuel Orozco Camacho, Stefan Horoi, Guy Wolf +1
Combining multiple machine learning models has long been a technique for enhancing performance, particularly in distributed settings. Traditional approaches, such as model ensemble…