Can Model Merging Improve Aggregation in DiLoCo?

Stefan Horoi, Benjamin Thérien, Guy Wolf +1

Model merging techniques, which aggregate independently finetuned models into one to combine their capabilities, have become a topic of significant interest in recent years, with a…