12 citations · 28 across the 9 of their papers we have counts for
3 papers · 1 filter
ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning
Shachar Don-Yehiya, Elad Venezian, Colin Raffel +3
We propose a new paradigm to continually evolve pretrained models, denoted ColD Fusion. It provides the benefits of multitask learning but leverages distributed computation with li…
Where to start? Analyzing the potential value of intermediate models
Leshem Choshen, Elad Venezian, Shachar Don-Yehia +2
Previous studies observed that finetuned models may be better base models than the vanilla pretrained model. Such a model, finetuned on some source dataset, may provide a better st…
Fusing finetuned models for better pretraining
Leshem Choshen, Elad Venezian, Noam Slonim +1
Pretrained models are the standard starting point for training. This approach consistently outperforms the use of a random initialization. However, pretraining is a costly endeavou…