1 citations · 1 across the 4 of their papers we have counts for
4 papers
CoRun: Padding is Simple and Efficient for Deterministic LLM Inference
Shiju Zhao, Jiacheng Yang, Qihang Chen +4
Despite fixed sampling parameters and random seeds, Large Language Model (LLM) inference exhibits output inconsistency, which undermines downstream tasks such as model evaluation a…
You Need an Encoder for Native Position-Independent Caching
Shiju Zhao, Junhao Hu, Jiaqi Zheng +1
The Key-Value (KV) cache of Large Language Models (LLMs) is prefix-based, making it highly inefficient for processing contexts retrieved in arbitrary order. Position-Independent Ca…
Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage
Junhao Hu, Fangze Li, Mingtao Xu +9
Large language models (LLMs) demonstrate strong capabilities across a wide range of complex tasks and are increasingly deployed at scale, placing significant demands on inference e…
MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Serving
Shiju Zhao, Junhao Hu, Rongxiao Huang +2
The context caching technique is employed to accelerate the Multimodal Large Language Model (MLLM) inference by prevailing serving platforms currently. However, this approach merel…