FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models

Zhongyu Zhao, Menghang Dong, Rongyu Zhang +6

Recent research has demonstrated that Feed-Forward Networks (FFNs) in Large Language Models (LLMs) play a pivotal role in storing diverse linguistic and factual knowledge. Conventi…