Routing in Sparsely-gated Language Models responds to Context

Stefan Arnold, Marian Fietta, Dilara Yesilbas

Language Models (LMs) recently incorporate mixture-of-experts layers consisting of a router and a collection of experts to scale up their parameter count given a fixed computationa…