MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling

Rachel S. Y. Teo, Tan M. Nguyen

Large-scale pre-training of deep models, followed by fine-tuning them, has become the cornerstone of natural language processing (NLP). The prevalence of data coupled with computat…