4 papers
Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start
Xueshen Liu, Yongji Wu, Yuncheng Yao +3
Modern LLM service providers increasingly rely on autoscaling and parallelism reconfiguration to respond to rapidly changing workloads, but cold-start latency remains a major bottl…
RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs
Yongji Wu, Xueshen Liu, Haizhong Zheng +5
Reinforcement learning (RL) has become essential for unlocking advanced reasoning capabilities in large language models (LLMs). RL workflows involve interleaving rollout and traini…
LEANN: A Low-Storage Vector Index
Yichuan Wang, Zhifei Li, Shu Liu +10
Embedding-based vector search underpins many important applications, such as recommendation and retrieval-augmented generation (RAG). It relies on vector indices to enable efficien…
An Extensible Software Transport Layer for GPU Networking
Yang Zhou, Zhongjie Chen, Ziming Mao +11
Fast-evolving machine learning (ML) workloads have increasing requirements for networking. However, host network transport on RDMA NICs is hard to evolve, causing problems for ML w…