8 citations · 12 across the 8 of their papers we have counts for
4 papers · 1 filter
Towards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache
Hanchen Li, Yuhan Liu, Yihua Cheng +2
Across large language model (LLM) applications, we observe an emerging trend for reusing KV caches to save the prefill delays of processing repeated input texts in different LLM in…
Loss-tolerant neural video codec aware congestion control for real time video communication
Zhengxu Xia, Hanchen Li, Junchen Jiang
Because of reinforcement learning's (RL) ability to automatically create more adaptive controlling logics beyond the hand-crafted heuristics, numerous effort has been made to apply…
Eloquent: A More Robust Transmission Scheme for LLM Token Streaming
Hanchen Li, Yuhan Liu, Yihua Cheng +3
To render each generated token in real-time for users, the Large Language Model (LLM) server generates tokens one by one and streams each token (or group of a few tokens) through t…
CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving
Yuhan Liu, Hanchen Li, Yihua Cheng +11
As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge. Yet using long contexts is challenging,…