1 citations · 1 across the 4 of their papers we have counts for
4 papers
CIERA: Cross-Iteration Exponent Reuse for Lossless Allgather in Sharded MoE Training
Ali Zafar Sadiq, Haiying Shen, Masahiro Tanaka
In training Mixture-of-Experts (MoE) models, sharded data parallelism partitions each expert's parameters across GPUs, requiring an Allgather operation to reconstruct the full weig…
PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
Bing Xie, Zhipeng Wang, Masahiro Tanaka +2
We present PEEK, a lightweight scheduling and eviction framework for both online (streaming) and offline (batch) LLM serving; this paper focuses on the online regime. PEEK maintain…
DeepCompile: A Compiler-Driven Approach to Optimizing Distributed Deep Learning Training
Masahiro Tanaka, Du Li, Umesh Chand +3
The rapid growth of deep learning models has increased the demand for efficient distributed training strategies. Fully sharded approaches like ZeRO-3 and FSDP partition model param…
Mitigating KV Cache Competition to Enhance User Experience in LLM Inference
Haiying Shen, Tanmoy Sen, Masahiro Tanaka
In Large Language Model (LLM) serving, the KV-cache (KVC) bottleneck causes high tail Time-to-First-Token (TTFT) and Time-Between-Tokens (TBT), impairing user experience, particula…