RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

Anthony. Lui, Mohamed. Elsaied, N. P. Savani

Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight…