4 papers
KVBuffer: IO-aware Serving for Linear Attention
Longwei Zou, Lin Zhong
Linear attention has recently gained significant attention for long-context inference due to its constant decoding cost with respect to context length. However, existing serving sy…
InstCache: A Predictive Cache for LLM Serving
Longwei Zou, Yan Liu, Jiamu Kang +3
The revolutionary capabilities of Large Language Models (LLMs) are attracting rapidly growing popularity and leading to soaring user requests to inference serving systems. Caching…
A Multi-Level Framework for Accelerating Training Transformer Models
Longwei Zou, Han Zhang, Yangdong Deng
The fast growing capabilities of large-scale deep learning models, such as Bert, GPT and ViT, are revolutionizing the landscape of NLP, CV and many other domains. Training such mod…
CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers
Longwei Zou, Qingyang Wang, Han Zhao +3
The fast-growing large scale language models are delivering unprecedented performance on almost all natural language processing tasks. However, the effectiveness of large language…