From Dense to Dynamic: Token-Difficulty Driven MoEfication of Pre-Trained LLMs

Kumari Nishu, Sachin Mehta, Samira Abnar +6

Training large language models (LLMs) for different inference constraints is computationally expensive, limiting control over efficiency-accuracy trade-offs. Moreover, once trained…