4 papers
PrefixFlow: Training-Time KV Caching for Schedule-Level Prefix Reuse in LLM RL Training
Pengbo Li, Feiyuan Zhang, Guangming Sheng +8
In reinforcement learning (RL) post-training, one prompt fans out into trajectories, yet standard trainers replay the shared prefix's forward and backward pass times; in lo…
Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices
Yuxiang Huang, Binhang Yuan, Xu Han +2
Scaling the input context length of a large language model (LLM) incurs a significant increase in computation cost and memory footprint to maintain the attention key-value (KV) cac…
DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inference
Jinwei Yao, Kaiqi Chen, Kexun Zhang +4
Large language models (LLMs) are increasingly employed for complex tasks that process multiple generation calls in a tree structure with shared prefixes of tokens, including few-sh…
LoHan: Low-Cost High-Performance Framework to Fine-Tune 100B Model on a Consumer GPU
Changyue Liao, Mo Sun, Zihan Yang +5
Nowadays, AI researchers become more and more interested in fine-tuning a pre-trained LLM, whose size has grown to up to over 100B parameters, for their downstream tasks. One appro…