Sparse Upcycling: Inference Inefficient Finetuning

Sasha Doubov, Nikhil Sardana, Vitaliy Chiley

Small, highly trained, open-source large language models are widely used due to their inference efficiency, but further improving their quality remains a challenge. Sparse upcyclin…