Eloquent: A More Robust Transmission Scheme for LLM Token Streaming

Abstract

To render each generated token in real-time for users, the Large Language Model (LLM) server generates tokens one by one and streams each token (or group of a few tokens) through the network to the user right after generation, which we refer to as LLM token streaming. However, under unstable network conditions, the LLM token streaming experience could suffer greatly from stalls since one packet loss could block the rendering of later tokens even if the packets containing them arrive on time. With a measurement study, we show that current applications suffer from increased stalls under unstable networks. For this emerging token streaming problem in LLM Chatbots that differs from previous multimedia and text applications, we propose a novel transmission scheme, called Eloquent, which puts newly generated tokens as well as currently unacknowledged tokens in the next outgoing packet. This ensures that each packet contains some new tokens and, in the meantime, is independently rendered when received, avoiding the aforementioned stalls caused by missing packets. Through simulation under various networks, we show Eloquent reduces stall ratio (proportion of token rendering wait time) by 71.0% compared to the retransmission method commonly used by real chatbot applications and by 31.6% compared to the baseline packet duplication scheme. By tailoring Eloquent to fit the token-by-token generation of LLM, we enable the Chatbots to respond like an eloquent speaker for users to better enjoy pervasive AI.

In SIGCOMM Workshop on Networks for AI Computing (NAIC '24)

References in corpus (8)

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Fast Inference from Transformers via Speculative Decoding
• Fast Distributed Inference Serving for Large Language Models
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services
• GRACE: Loss-Resilient Real-Time Video through Neural Codecs

Cited by in corpus (1)

• Streaming, Fast and Slow: Cognitive Load-Aware Streaming for Efficient LLM Serving