16 citations · 21 across the 14 of their papers we have counts for
16 papers
mKernel: Fast Multi-GPU, Multi-Node Fused Kernels
Ziming Mao, Yihan Zhang, Shawn Wei Chew +5
Communication has become a bottleneck in distributed training and inference of large models. Overlapping communication with computation at the granularity of kernels, on separate s…
Reality Is the Final Verifier: On Two Key Gaps in Agentic Software Engineering
Alexander Krentsel, Shubham Agarwal, Mert Cemri +5
Software development follows an implementation-verification loop in which developers or agents iteratively revise an implementation until an evaluator, such as a test suite, accept…
CommBench: Can LLMs Write Correct and Efficient GPU Communication Code?
Shuang Ma, Yuyi Li, Yihan Zhang +12
Training and serving large language models (LLMs) rely heavily on high-performance GPU communication, yet implementing efficient GPU communication primitives requires deep expertis…
The Time is Here for Just-in-Time Systems: Challenges and Opportunities
Shu Liu, Alexander Krentsel, Shubham Agarwal +8
Core systems like key-value stores have historically taken years to build, and are designed to be general so as to amortize cost across deployments, paying a significant performanc…
UCCL-Zip: Lossless Compression Supercharged GPU Communication
Shuang Ma, Chon Lam Lao, Zhiying Xu +8
The rapid growth of large language models (LLMs) has made GPU communication a critical bottleneck. While prior work reduces communication volume via quantization or lossy compressi…
K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Model
Shiyi Cao, Ziming Mao, Joseph E. Gonzalez +1
Optimizing GPU kernels is critical for efficient modern machine learning systems yet remains challenging due to the complex interplay of design factors and rapid hardware evolution…
UCCL-EP: Portable Expert-Parallel Communication
Ziming Mao, Yihan Zhang, Chihan Cui +9
Mixture-of-Experts (MoE) workloads rely on expert parallelism (EP) to achieve high GPU efficiency. State-of-the-art EP communication systems such as DeepEP demonstrate strong perfo…
SkyNomad: On Using Multi-Region Spot Instances to Minimize AI Batch Job Cost
Zhifei Li, Tian Xia, Ziming Mao +9
AI batch jobs such as model training, inference pipelines, and data analytics require substantial GPU resources and often need to finish before a deadline. Spot instances offer 3-1…
LEANN: A Low-Storage Vector Index
Yichuan Wang, Zhifei Li, Shu Liu +10
Embedding-based vector search underpins many important applications, such as recommendation and retrieval-augmented generation (RAG). It relies on vector indices to enable efficien…
SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inference
Tian Xia, Ziming Mao, Jamison Kerney +5
Serving Large Language Models (LLMs) efficiently in multi-region setups remains a challenge. Due to cost and GPU availability concerns, providers typically deploy LLMs in multiple…
An Extensible Software Transport Layer for GPU Networking
Yang Zhou, Zhongjie Chen, Ziming Mao +11
Fast-evolving machine learning (ML) workloads have increasing requirements for networking. However, host network transport on RDMA NICs is hard to evolve, causing problems for ML w…
Locality-aware Fair Scheduling in LLM Serving
Shiyi Cao, Yichuan Wang, Ziming Mao +10
Large language model (LLM) inference workload dominates a wide variety of modern AI applications, ranging from multi-turn conversation to document analysis. Balancing fairness and…
Ray Data: A dynamic and scalable system for heterogeneous data processing
Frank Sifei Luan, Ron Yifeng Wang, Yile Gu +12
While ML model training and inference are both GPU-intensive, CPU-based data processing is often the bottleneck. Existing distributed data processing systems assume homogeneous res…
Revisiting Cache Freshness for Emerging Real-Time Applications
Ziming Mao, Rishabh Iyer, Scott Shenker +1
Caching is widely used in industry to improve application performance by reducing data-access latency and taking the load off the backend infrastructure. TTLs have become the de-fa…
Pie: Pooling CPU Memory for LLM Inference
Yi Xu, Ziming Mao, Xiangxi Mo +2
The rapid growth of LLMs has revolutionized natural language processing and AI analysis, but their increasing size and memory demands present significant challenges. A common solut…
SkyServe: Serving AI Models across Regions and Clouds with Spot Instances
Ziming Mao, Tian Xia, Zhanghao Wu +6
Recent years have witnessed an explosive growth of AI models. The high cost of hosting AI services on GPUs and their demanding service requirements, make it timely and challenging…