Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling

Rohan Shravan

This paper reports on training a hundred-billion-parameter sparse mixture of experts on a single eight-GPU node, end to end. LightningLM 0.1V is a recurrence-backbone language mode…