4 papers
On-Policy Attention Linearization
Arian Raje, Anupam Nayak, Anthony Fei +3
Hybrid transformer architectures that replace most softmax attention layers with linear attention offer transformer-level quality at a fraction of the memory cost. Rather than pret…
An infinite family of doubly saturated -good graphs
Abhishek Saigal, Akaash R. Parthasarathy
For every odd integer , we prove that an explicit circulant graph on vertices is doubly saturated -good. The graph is triangle-free and has independence num…
Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM pools
Shivam Patel, Akaash R. Parthasarathy, Ankur Mallick +2
Large language model (LLM) query routers typically assign each query to a suitable model that balances response accuracy and monetary cost. Subsequently, separate system-level load…
PithTrain: A Compact and Agent-Native MoE Training System
Ruihang Lai, Hao Kang, Haozhan Tang +6
Mixture-of-Experts (MoE) has become the dominant architecture for frontier language models. To meet this demand, production frameworks have built optimized MoE training stacks over…