5 papers
DynSplit-KV: Dynamic Semantic Splitting for KVCache Compression in Efficient Long-Context LLM Inference
Jiancai Ye, Jun Liu, Qingchen Li +5
Although Key-Value (KV) Cache is essential for efficient large language models (LLMs) inference, its growing memory footprint in long-context scenarios poses a significant bottlene…
SpeContext: Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMs
Jiaming Xu, Jiayi Pan, Hanzhen Wang +4
In this paper, we point out that the objective of the retrieval algorithms is to align with the LLM, which is similar to the objective of knowledge distillation in LLMs. We analyze…
BalanceGS: Algorithm-System Co-design for Efficient 3D Gaussian Splatting Training on GPU
Junyi Wu, Jiaming Xu, Jinhao Li +4
3D Gaussian Splatting (3DGS) has emerged as a promising 3D reconstruction technique. The traditional 3DGS training pipeline follows three sequential steps: Gaussian densification,…
SpecDiff: Accelerating Diffusion Model Inference with Self-Speculation
Jiayi Pan, Jiaming Xu, Yongkang Zhou +1
Feature caching has recently emerged as a promising method for diffusion model acceleration. It effectively alleviates the inefficiency problem caused by high computational require…
SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting
Jiaming Xu, Jiayi Pan, Yongkang Zhou +5
Early exiting has recently emerged as a promising technique for accelerating large language models (LLMs) by effectively reducing the hardware computation and memory access. In thi…