11 papers
Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning
Qiyuan Zhu, Dezhi Li, Pengyu Cheng +8
Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inf…
Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models
Hao Wang, Hao Gu, Hongming Piao +6
The paper introduces CurioSFT, an entropy-preserving supervised fine-tuning approach that uses adaptive self-distillation to keep exploration abilities in large reasoning models, l…
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Xiang Hu, Xinyu Wei, Hao Gu +10
Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention. Chunk-wise sparse atten…
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
Xintong Yang, Hao Gu, Binxing Xu +6
Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, qui…
AutoSG: LLM-Driven Solver Generation Solely from Task Prompts for Expensive Optimization
Haoran Gu, Handing Wang, Yi Mei +1
Expensive optimization tasks are ubiquitous in real-world applications, demanding highly specialized solvers. While LLM-driven automated solver generation shows promise, current pa…
Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs
Binxing Xu, Hao Gu, Lujun Li +8
Training LLMs at ultra-low precision remains a formidable challenge. Direct low-bit QAT often suffers from convergence instability and substantial training costs, exacerbated by qu…