Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts

Kwanhee Lee, Namhoon Lee, Dan Alistarh

Mixture-of-Experts (MoE) architectures allow frontier language models to scale to trillions of parameters, but their deployment is constrained by massive memory footprints and memo…