MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference

Abstract

Wafer-scale accelerators promise ultra-low-latency AI inference, but current system stacks still carry an unsustainably high cost premium. The reason is that many current and emerging inference techniques, such as batching and MoE, introduce runtime dynamism that existing wafer-scale systems cannot support efficiently. As a result, they must choose between two unsatisfactory options: accept suboptimal latency, or preserve low latency by restricting dynamic techniques, thereby limiting the models they can support and reducing aggregate throughput. This tradeoff significantly increases the cost of low-latency serving. We present MeshRT, the first wafer-scale system to achieve both ultra-low latency and high throughput for LLM inference. MeshRT achieves this via a novel system architecture, which we call compile-time governance of runtime dynamism. This architecture features new system abstractions and mechanisms that enable MeshRT to compile dynamic events arising from LLM inference directly into per-core schedules, enabling asynchronous and decentralized execution with low runtime overhead. Implemented on a commodity wafer-scale accelerator, MeshRT preserves ultra-low latency while improving throughput by up to 56$\times$ over the state-of-the-art system for wafer-scale accelerators. Against measured single-instance SGLang results on H200, MeshRT achieves 4.5-7.0$\times$ lower latency. Under ideal linear scaling of those GPU measurements to an approximately power-matched H200 cluster, MeshRT achieves 1.7-13.0$\times$ higher decode system throughput in the low-latency regime.

arXiv commentAccepted to the 32nd ACM Symposium on Operating Systems Principles (SOSP 2026)

References in corpus (11)

• Generating Long Sequences with Sparse Transformers
• Mixtral of Experts
• SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification
• Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward
• Dissecting the Graphcore IPU Architecture via Microbenchmarking
• SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts
• Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor with T10
• Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
• OpenThoughts: Data Recipes for Reasoning Models
• SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference
• Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batching