24 citations · 43 across the 31 of their papers we have counts for
32 papers
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Yi Pan, Haocheng Xi, Kan Zhu +10
Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). Although they comp…
Presage: Prefetch Search via Agent-Guided Experiments
Matthew Giordano, Parthasarathy Ranganathan, Baris Kasikci +1
Data prefetching is an established technique to mitigate cache miss latency and keep the processor saturated with data. Software exists in a unique position to issue prefetches, ha…
Themis: Software-Defined Hardware Prefetching
Keisuke Kamahori, Neil Adit, Kan Zhu +13
Data cache misses represent a significant portion of stall cycles in datacenter workloads. Hardware prefetchers that reduce such stalls by fetching data ahead of time have become i…
TraceLab: Characterizing Coding Agent Workloads for LLM Serving
Kan Zhu, Mathew Jacob, Chenxi Ma +4
Coding agents are rapidly becoming a major application of agentic LLMs, but serving them efficiently remains challenging. Progress on this challenge requires understanding real wor…
M*: A Modular, Extensible, Serving System for Multimodal Models
Atindra Jha, Naomi Sagan, Keisuke Kamahori +9
We are entering a new era of composite model architectures that integrate diverse components such as vision encoders, language backbones, diffusion and flow heads, audio codecs, ac…
Ekka: Automated Diagnosis of Silent Errors in LLM Inference
Yile Gu, Zhen Zhang, Shaowei Zhu +4
LLM serving frameworks are quickly evolving with a complex software stack and a vast number of optimizations. The rapid development process can introduce silent errors where output…
MURMUR: An Efficient Inference System for Long-Form ASR
Wei-Tzu Lee, Keisuke Kamahori, Baris Kasikci
Long-form automatic speech recognition (ASR) requires both high accuracy and low latency, but existing systems force a trade-off between the two. Chunk-based pipelines process audi…
DynaFlow: Transparent and Flexible Intra-Device Parallelism via Programmable Operator Scheduling
Yi Pan, Yile Gu, Jinbin Luo +7
Intra-device parallelism addresses resource under-utilization in ML inference and training by overlapping the execution of operators with different resource usage. However, its wid…
Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP
Yilong Zhao, Xiaonan Nie, Kan Zhu +6
Context parallelism (CP) has been widely adopted to support the growing context length in foundation model pretraining. However, existing designs fail to handle the large variation…
VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?
Keisuke Kamahori, Shihang Li, Simon Peter +1
For years, we have built LLM serving systems like any other critical infrastructure, with a single general-purpose stack hand-tuned over many engineer-years to support every model…
VoxServe: Streaming-Centric Serving System for Speech Language Models
Keisuke Kamahori, Wei-Tzu Lee, Atindra Jha +4
Deploying modern Speech Language Models (SpeechLMs) in streaming settings requires systems that provide low latency, high throughput, and strong guarantees of streamability. Existi…
Magneton: Optimizing Energy Efficiency of ML Systems via Differential Energy Debugging
Yi Pan, Wenbo Qian, Dedong Xie +3
The training and deployment of machine learning (ML) models have become extremely energy-intensive. While existing optimization efforts focus primarily on hardware energy efficienc…
Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding
Yilong Zhao, Jiaming Tang, Kan Zhu +10
Reasoning language models have demonstrated remarkable capabilities on challenging tasks by generating elaborate chain-of-thought (CoT) solutions. However, such lengthy generation…
AgentFlux: Decoupled Fine-Tuning & Inference for On-Device Agentic Systems
Rohan Kadekodi, Zhan Jin, Keisuke Kamahori +5
The deployment of Large Language Models (LLMs) as agentic orchestrators has revolutionized task automation, but the need for privacy-preserving, cost-effective solutions demands on…
Jenga: Responsive Tiered Memory Management without Thrashing
Rohan Kadekodi, Haoran Peng, Gilbert Bernstein +2
A heterogeneous memory has a single address space with fast access to some addresses (a fast tier of DRAM) and slow access to other addresses (a capacity tier of CXL-attached memor…
PolyServe: Efficient Multi-SLO Serving at Scale
Kan Zhu, Haiyang Shi, Le Xu +4
Advances in Large Language Models (LLMs) have led to a surge of LLM-powered applications. These applications have diverse token-generation latency requirements. As a result, simply…
gigiProfiler: Diagnosing Performance Issues by Uncovering Application Resource Bottlenecks
Yigong Hu, Haodong Zheng, Yicheng Liu +3
Diagnosing performance bottlenecks in modern software is essential yet challenging, particularly as applications become more complex and rely on custom resource management policies…
ConsumerBench: Benchmarking Generative AI Applications on End-User Devices
Yile Gu, Rohan Kadekodi, Hoang Nguyen +3
The recent shift in Generative AI (GenAI) applications from cloud-only environments to end-user devices introduces new challenges in resource management, system efficiency, and use…
Fake Runs, Real Fixes -- Analyzing xPU Performance Through Simulation
Ioannis Zarkadas, Amanda Tomlinson, Asaf Cidon +2
As models become larger, ML accelerators are a scarce resource whose performance must be continually optimized to improve efficiency. Existing performance analysis tools are coarse…
Scalable and Accurate Application-Level Crash-Consistency Testing via Representative Testing
Yile Gu, Ian Neal, Jiexiao Xu +7
Crash consistency is essential for applications that must persist data. Crash-consistency testing has been commonly applied to find crash-consistency bugs in applications. The cras…
Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
Kan Zhu, Tian Tang, Qinyu Xu +7
Long-context models are essential for many applications but face inefficiencies in loading large KV caches during decoding. Prior methods enforce fixed token budgets for sparse att…
TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval
Chien-Yu Lin, Keisuke Kamahori, Yiyu Liu +11
Retrieval-augmented generation (RAG) extends large language models (LLMs) with external data sources to enhance factual correctness and domain coverage. Modern RAG pipelines rely o…
LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximation
Keisuke Kamahori, Jungo Kasai, Noriyuki Kojima +1
Modern automatic speech recognition (ASR) models, such as OpenAI's Whisper, rely on deep encoder-decoder architectures, and their encoders are a critical bottleneck for efficient d…
Argos: Agentic Time-Series Anomaly Detection with Autonomous Rule Generation via Large Language Models
Yile Gu, Yifan Xiong, Jonathan Mace +4
Observability in cloud infrastructure is critical for service providers, driving the widespread adoption of anomaly detection systems for monitoring metrics. However, existing syst…