Mixture of Tokens: Continuous MoE through Cross-Example Aggregation

Szymon Antoniak, Michał Krutul, Maciej Pióro +7

Mixture of Experts (MoE) models based on Transformer architecture are pushing the boundaries of language and vision tasks. The allure of these models lies in their ability to subst…