Mixture-of-Kittens: MoE Megakernel for NVL72s

Stuart H. Sul, Nash Brown, Henry Wildermuth +3

AI accelerator systems are rapidly consolidating into scale-up architectures, where tens to thousands of GPUs communicate over high-bandwidth, single-hop fabrics. We find that exis…