2 citations · 4 across the 13 of their papers we have counts for
19 papers
Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference
Zhenxu Tian, Kebin Liu, Zhengwu Yang +8
Long-context decoding in large language models (LLMs) is increasingly bottlenecked by the memory and computation required to attend over an ever-growing Key-Value (KV) cache at eac…
Virtual Head Attention
Xibo Ding, Guoxia Wang, Jinle Zeng +4
The design of attention mechanisms in long-context large language models faces an ``impossible triangle'' among KV cache size, training and prefill FLOPs, and model quality. GQA ac…
FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention
Guoxia Wang, Qianyue He, Siming Wu +6
Long-context training of large language and multimodal models couples structured attention masks (document, prefix-LM, share-question, sliding-window, block-sparse) with multi-node…
CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding
Yulong Liu, Xiaotian Han, Junyuan Shang +6
Vision-language models face a fundamental scaling bottleneck: the number of visual tokens grows with both temporal duration and spatial resolution, making long-video understanding…
RESUME: Recurrent State Updates from Motion and Residual Signals for Efficient Video Language Modeling
Can Zhang, Xiaotian Han, Junyuan Shang +5
Existing video language models encode sampled RGB frames independently, so a long video must either exhaust the token budget or drop the changes between sampled frames. Codec-aware…
From Intent to Evidence: Policy-Steered Multi-Strategy Retrieval for Long-Video Agents
Can Zhang, Baofeng Zhang, Xiaotian Han +5
Existing long-video agents acquire evidence through one uniform behavior, ignoring whether the required evidence is concentrated, requires broad occurrence coverage, or must discri…
Aggregating Visual Information with Optimal Transport for VideoLM Token Compression
Wenti Yin, Xiaotian Han, Junyuan Shang +5
Video language models process videos as dense visual-token sequences with substantial representational redundancy. Compressing these sequences is therefore essential for reducing t…
Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Yehan Yang, Junyuan Shang, Yang Li +3
Long-context LLM inference is bottlenecked by quadratic attention computation and growing KV-cache costs. Existing sparse attention and KV-compression methods typically decide whic…
ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation
Yao Chen, Yinqi Yang, Junyuan Shang +6
Hybrid architectures combining full attention (FA) and sliding-window attention (SWA) are a promising paradigm for efficient LLM inference. However, existing methods typically rely…
PP-OCRv6: From 1.5M to 34.5M Parameters, Surpassing Billion-Scale VLMs on OCR Tasks
Yubo Zhang, Xueqing Wang, Manhui Lin +13
Vision-Language Models (VLMs) have achieved impressive results on general vision-language tasks, yet they suffer from hallucination, imprecise localization, and prohibitive computa…
Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuning
Yifei Cheng, Xianglin Yang, Guoxia Wang +5
Sharpness-aware minimization (SAM) seeks the minima with a flat loss landscape to improve the generalization performance in machine learning tasks, including fine-tuning. However,…
RRAttention: Dynamic Block Sparse Attention via Per-Head Round-Robin Shifts for Long-Context Inference
Siran Liu, Guoxia Wang, Sa Wang +7
The quadratic complexity of attention mechanisms poses a critical bottleneck for large language models processing long contexts. While dynamic sparse attention methods offer input-…
ERNIE 5.0 Technical Report
Haifeng Wang, Hua Wu, Tian Wu +432
In this report, we introduce ERNIE 5.0, a natively autoregressive foundation model desinged for unified multimodal understanding and generation across text, image, video, and audio…
Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities
Zhiwei Hao, Jianyuan Guo, Li Shen +6
Large language models (LLMs) have achieved impressive performance across various domains. However, the substantial hardware resources required for their training present a signific…
SeWA: Selective Weight Average via Probabilistic Masking
Peng Wang, Shengchao Hu, Zerui Tao +5
Weight averaging has become a standard technique for enhancing model performance. However, methods such as Stochastic Weight Averaging (SWA) and Latest Weight Averaging (LAWA) ofte…
AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping
Guoxia Wang, Shuai Li, Congliang Chen +5
Loss spikes remain a persistent obstacle in large-scale language model pretraining. While previous research has attempted to identify the root cause of loss spikes by investigating…
FlashMask: Efficient and Rich Mask Extension of FlashAttention
Guoxia Wang, Jinle Zeng, Xiyuan Xiao +7
The computational and memory demands of vanilla attention scale quadratically with the sequence length , posing significant challenges for processing long sequences in Transform…
NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time
Yilong Chen, Guoxia Wang, Junyuan Shang +7
Large Language Models (LLMs) have ignited an innovative surge of AI applications, marking a new era of exciting possibilities equipped with extended context windows. However, hosti…
A Framework for Cost-Effective and Self-Adaptive LLM Shaking and Recovery Mechanism
Zhiyu Chen, Yu Li, Suochao Zhang +4
As Large Language Models (LLMs) gain great success in real-world applications, an increasing number of users are seeking to develop and deploy their customized LLMs through cloud s…