CORTEX: Learning to Share and Specialize in Dense Language Models

Chuiyang Meng, Ming Tang, Vincent W. S. Wong

Large language models are trained on heterogeneous data mixtures, where different knowledge domains require both shared knowledge and specialization. Existing modular approaches ty…