5 papers
Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput
Jingwei Song, Wanyi Chen, Xinyuan Song +7
Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens that are later verified by a stronger target model. While…
Lattica: A Decentralized Cross-NAT Communication Framework for Scalable AI Inference and Training
Ween Yang, Jason Liu, Suli Wang +4
The rapid expansion of distributed Artificial Intelligence (AI) workloads beyond centralized data centers creates a demand for new communication substrates. These substrates must o…
Parallax: Efficient LLM Inference Service over Decentralized Environment
Chris Tong, Youhe Jiang, Gufeng Chen +6
Deploying a large language model (LLM) inference service remains costly because centralized serving depends on specialized GPU clusters and high-bandwidth interconnects in datacent…
VeriLLM: A Lightweight Framework for Publicly Verifiable Decentralized Inference
Ke Wang, Zishuo Zhao, Xinyuan Song +7
Decentralized inference provides a scalable and resilient paradigm for serving large language models (LLMs), enabling fragmented global resource utilization and reducing reliance o…
Echo: Decoupling Inference and Training for Large-Scale RL Alignment on Heterogeneous Swarms
Jie Xiao, Changyuan Fan, Qingnan Ren +6
Modern RL-based post-training for large language models (LLMs) co-locate trajectory sampling and policy optimisation on the same GPU cluster, forcing the system to switch between i…