4 papers
LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval
Zhenyu Ning, Guangda Liu, Qihao Jin +4
Recent developments in Video Large Language Models (Video LLMs) have enabled models to process hour-long videos and exhibit exceptional performance. Nonetheless, the Key-Value (KV)…
FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference
Guangda Liu, Chengwei Li, Zhenyu Ning +5
Large language models (LLMs) are widely deployed with rapidly expanding context windows to support increasingly demanding applications. However, long contexts pose significant depl…
Inf-MLLM: Efficient Streaming Inference of Multimodal Large Language Models on a Single GPU
Zhenyu Ning, Jieru Zhao, Qihao Jin +2
Multimodal Large Language Models (MLLMs) are distinguished by their multimodal comprehensive ability and widely used in many real-world applications including GPT-4o, autonomous dr…
The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving
Pai Zeng, Zhenyu Ning, Jieru Zhao +5
We survey the large language model (LLM) serving area to understand the intricate dynamics between cost-efficiency and accuracy, which is magnified by the growing need for longer c…