works on
continual learning 1large language models 1modular networks 1parameter-efficient fine-tuning 1sparse routing 1

From the 1 of 1 linked paper with an AI index.

collaborators

1 paper

cs.LG2026

Mixtures of SubExperts for Large Language Continual Learning

Haeyong Kang, Hee Suk Yoon, Dahua Feng +1

The paper proposes Mixtures of SubExperts (MoSEs), a modular and sparse extension to transformer layers that uses lightweight sub-modules and a learned routing function to let larg…