Showing 2024Show all
3 papers · 1 filter
cs.LG2024
OneAdapt: Fast Configuration Adaptation for Video Analytics Applications via Backpropagation
Kuntai Du, Yuhan Liu, Yitian Hao +5
Deep learning inference on streaming media data, such as object detection in video or LiDAR feeds and text extraction from audio waves, is now ubiquitous. To achieve high inference…
cs.LG2024
LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts
Zhuohan Gu, Jiayi Yao, Kuntai Du +1
As large language models (LLMs) show impressive performance on complex tasks, they still struggle with longer contextual understanding and high computational costs. To balance effi…
cs.NI2024
CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving
Yuhan Liu, Hanchen Li, Yihua Cheng +11
As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge. Yet using long contexts is challenging,…