13 papers
CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding
Yulong Liu, Xiaotian Han, Junyuan Shang +6
Vision-language models face a fundamental scaling bottleneck: the number of visual tokens grows with both temporal duration and spatial resolution, making long-video understanding…
OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
Dezhi Li, Lujun Li, Qiyuan Zhu +4
Mixture-of-Experts (MoE) models enable efficient scaling of large language models but face critical deployment challenges due to massive memory requirements. Existing pruning metho…
Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning
Qiyuan Zhu, Dezhi Li, Pengyu Cheng +8
Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inf…
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Xiang Hu, Xinyu Wei, Hao Gu +10
Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention. Chunk-wise sparse atten…
Self-Evolving Deep Research via Joint Generation and Evaluation
Han Zhu, Chengkun Cai, Yuanfeng Song +3
Large Language Models (LLMs) have become increasingly adopted in daily applications, with deep research standing out as a particularly important capability. Unlike traditional ques…
RW-TTT: Batched Serving for Request-Owned Test-Time Training State
Jian Yang, Zhizhuo Kou, Yao Tian +4
Test-time training (TTT) adapts an LLM during generation by reading and updating request-owned state, such as fast weights, low-rank deltas, or streaming learner state. This breaks…
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
Xintong Yang, Hao Gu, Binxing Xu +6
Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, qui…
When Text Hijacks Vision: Benchmarking and Mitigating Text Overlay-Induced Hallucination in Vision Language Models
Cui Yakun, Xingqun Qi, TianTian Geng +3
Recent advances in Vision-Language Models (VLMs) have substantially enhanced their ability across multimodal video understanding benchmarks spanning temporal, action, object, and s…
Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs
Binxing Xu, Hao Gu, Lujun Li +8
Training LLMs at ultra-low precision remains a formidable challenge. Direct low-bit QAT often suffers from convergence instability and substantial training costs, exacerbated by qu…
QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch
Hao Gu, Hao Wang, Jiacheng Liu +9
Large language model (LLM) reinforcement learning (RL) pipelines are often bottlenecked by rollout generation, making end-to-end training slow. Recent work mitigates this by runnin…
Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging
Lujun Li, Zhu Qiyuan, Jiacheng Wang +4
Mixture of Experts (MoE) LLMs face significant obstacles due to their massive parameter scale, which imposes memory, storage, and deployment challenges. Although recent expert merg…
BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebook
Hao Gu, Lujun Li, Hao Wang +7
Binary quantization represents the most extreme form of compression, reducing weights to +/-1 for maximal memory and computational efficiency. While recent sparsity-aware binarizat…
Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuning
Yue Ma, Yulong Liu, Qiyuan Zhu +8
Recently, breakthroughs in the video diffusion transformer have shown remarkable capabilities in diverse motion generations. As for the motion-transfer task, current methods mainly…