4 citations · 15 across the 21 of their papers we have counts for
22 papers
Supercritical sharpness for Ising double random currents
Trishen S. Gunaratnam, Christoforos Panagiotis, Romain Panis +1
We show that the percolation model induced by the double random current representation of the Ising model on , with , satisfies local uniqueness of the macros…
StateFlow: Sequence Pipeline Parallelism for Long-Context Modeling with Linear Recurrence
Wenxuan Zhao, Yingfa Chen, Xu Han +7
Long-context training is increasingly important for large language models, and linear attention and state space models have become popular for improving long-context efficiency. Ho…
Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
Haodong Wen, Yiran Zhang, Yingfa Chen +1
While large language models (LLMs) can solve advanced reasoning problems in seconds, we show that even frontier models fail to perform a much simpler operation: exactly copying an…
Rethinking the Role of Efficient Attention in Hybrid Architectures
Ziqing Qiao, Yinuo Xu, Chaojun Xiao +6
Modern language models increasingly adopt hybrid architectures that combine full attention with efficient attention modules, such as sliding-window attention (SWA) and recurrent se…
Attention Amnesia in Hybrid LLMs: When CoT Fine-Tuning Breaks Long-Range Recall, and How to Fix It
Xinyu Zhou, Boyu Zhu, Yi Xu +4
Chain-of-thought (CoT) supervised fine-tuning (SFT) is widely adopted to improve reasoning ability, yet we find that it systematically degrades long-context recall in hybrid linear…
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
Chenyang Song, Weilin Zhao, Xu Han +3
While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-acc…
Student-in-the-Loop Chain-of-Thought Distillation via Generation-Time Selection
Chaoqun He, Yingfa Chen, Chaojun Xiao +2
Large reasoning models achieve strong performance on complex tasks through long chain-of-thought (CoT) trajectories, but directly transferring such reasoning processes to smaller m…
MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
MiniCPM Team, Wenhao An, Yingfa Chen +44
The evolution of large language models (LLMs) towards applications with ultra-long contexts faces challenges posed by the high computational and memory costs of the Transformer arc…
Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts
Yingfa Chen, Zhen Leng Thai, Zihan Zhou +6
Hybrid Transformer architectures, which combine softmax attention blocks and recurrent neural networks (RNNs), have shown a desirable performance-throughput tradeoff for long-conte…
StateX: Enhancing RNN Recall via Post-training State Expansion
Xingyu Shen, Yingfa Chen, Zhen Leng Thai +3
Recurrent neural networks (RNNs), such as linear attention and state-space models, have gained popularity due to their constant per-token complexity when processing long contexts.…
BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
Chenyang Song, Weilin Zhao, Xu Han +5
To alleviate the computational burden of large language models (LLMs), architectures with activation sparsity, represented by mixture-of-experts (MoE), have attracted increasing at…
Cost-Optimal Grouped-Query Attention for Long-Context Modeling
Yingfa Chen, Yutong Wu, Chenyang Song +5
Grouped-Query Attention (GQA) is a widely adopted strategy for reducing the computational cost of attention layers in large language models (LLMs). However, current GQA configurati…
Sparsing Law: Towards Large Language Models with Greater Activation Sparsity
Yuqi Luo, Chenyang Song, Xu Han +7
Activation sparsity denotes the existence of substantial weakly-contributed elements within activation outputs that can be eliminated, benefiting many important applications concer…
Stuffed Mamba: Oversized States Lead to the Inability to Forget
Yingfa Chen, Xinrong Zhang, Shengding Hu +3
Recent advancements in recurrent architectures, such as Mamba and RWKV, have showcased strong language capabilities. Unlike transformer-based models, these architectures encode all…
Configurable Foundation Models: Building LLMs from a Modular Perspective
Chaojun Xiao, Zhengyan Zhang, Chenyang Song +20
Advancements in LLMs have recently unveiled challenges tied to computational efficiency and continual scalability due to their requirements of huge parameters, making the applicati…
Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scripts
Yingfa Chen, Chenlong Hu, Cong Feng +5
This study presents a multi-modal multi-granularity tokenizer specifically designed for analyzing ancient Chinese scripts, focusing on the Chu bamboo slip (CBS) script used during…
Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models
Xinrong Zhang, Yingfa Chen, Shengding Hu +6
As large language models (LLMs) increasingly permeate daily lives, there is a growing demand for real-time interactions that mirror human conversations. Traditional turn-based chat…
Robust and Scalable Model Editing for Large Language Models
Yingfa Chen, Zhengyan Zhang, Xu Han +6
Large language models (LLMs) can make predictions using parametric knowledge--knowledge encoded in the model weights--or contextual knowledge--knowledge presented in the context. I…
Bench: Extending Long Context Evaluation Beyond 100K Tokens
Xinrong Zhang, Yingfa Chen, Shengding Hu +8
Processing and reasoning over long contexts is crucial for many practical applications of Large Language Models (LLMs), such as document comprehension and agent construction. Despi…
CFDBench: A Large-Scale Benchmark for Machine Learning Methods in Fluid Dynamics
Yining Luo, Yingfa Chen, Zhen Zhang
In recent years, applying deep learning to solve physics problems has attracted much attention. Data-driven deep learning methods produce fast numerical operators that can learn ap…
READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises
Chenglei Si, Zhengyan Zhang, Yingfa Chen +3
For many real-world applications, the user-generated inputs usually contain various noises due to speech recognition errors caused by linguistic variations1 or typographical errors…
Sub-Character Tokenization for Chinese Pretrained Language Models
Chenglei Si, Zhengyan Zhang, Yingfa Chen +6
Tokenization is fundamental to pretrained language models (PLMs). Existing tokenization methods for Chinese PLMs typically treat each character as an indivisible token. However, th…