2 citations · 4 across the 13 of their papers we have counts for
13 papers · 1 filter
Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference
Zhenxu Tian, Kebin Liu, Zhengwu Yang +8
Long-context decoding in large language models (LLMs) is increasingly bottlenecked by the memory and computation required to attend over an ever-growing Key-Value (KV) cache at eac…
Virtual Head Attention
Xibo Ding, Guoxia Wang, Jinle Zeng +4
The design of attention mechanisms in long-context large language models faces an ``impossible triangle'' among KV cache size, training and prefill FLOPs, and model quality. GQA ac…
FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention
Guoxia Wang, Qianyue He, Siming Wu +6
Long-context training of large language and multimodal models couples structured attention masks (document, prefix-LM, share-question, sliding-window, block-sparse) with multi-node…
CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding
Yulong Liu, Xiaotian Han, Junyuan Shang +6
Vision-language models face a fundamental scaling bottleneck: the number of visual tokens grows with both temporal duration and spatial resolution, making long-video understanding…
RESUME: Recurrent State Updates from Motion and Residual Signals for Efficient Video Language Modeling
Can Zhang, Xiaotian Han, Junyuan Shang +5
Existing video language models encode sampled RGB frames independently, so a long video must either exhaust the token budget or drop the changes between sampled frames. Codec-aware…
From Intent to Evidence: Policy-Steered Multi-Strategy Retrieval for Long-Video Agents
Can Zhang, Baofeng Zhang, Xiaotian Han +5
Existing long-video agents acquire evidence through one uniform behavior, ignoring whether the required evidence is concentrated, requires broad occurrence coverage, or must discri…
Aggregating Visual Information with Optimal Transport for VideoLM Token Compression
Wenti Yin, Xiaotian Han, Junyuan Shang +5
Video language models process videos as dense visual-token sequences with substantial representational redundancy. Compressing these sequences is therefore essential for reducing t…
Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Yehan Yang, Junyuan Shang, Yang Li +3
Long-context LLM inference is bottlenecked by quadratic attention computation and growing KV-cache costs. Existing sparse attention and KV-compression methods typically decide whic…
ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation
Yao Chen, Yinqi Yang, Junyuan Shang +6
Hybrid architectures combining full attention (FA) and sliding-window attention (SWA) are a promising paradigm for efficient LLM inference. However, existing methods typically rely…
PP-OCRv6: From 1.5M to 34.5M Parameters, Surpassing Billion-Scale VLMs on OCR Tasks
Yubo Zhang, Xueqing Wang, Manhui Lin +13
Vision-Language Models (VLMs) have achieved impressive results on general vision-language tasks, yet they suffer from hallucination, imprecise localization, and prohibitive computa…
Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuning
Yifei Cheng, Xianglin Yang, Guoxia Wang +5
Sharpness-aware minimization (SAM) seeks the minima with a flat loss landscape to improve the generalization performance in machine learning tasks, including fine-tuning. However,…
RRAttention: Dynamic Block Sparse Attention via Per-Head Round-Robin Shifts for Long-Context Inference
Siran Liu, Guoxia Wang, Sa Wang +7
The quadratic complexity of attention mechanisms poses a critical bottleneck for large language models processing long contexts. While dynamic sparse attention methods offer input-…
ERNIE 5.0 Technical Report
Haifeng Wang, Hua Wu, Tian Wu +432
In this report, we introduce ERNIE 5.0, a natively autoregressive foundation model desinged for unified multimodal understanding and generation across text, image, video, and audio…