1 citations · 1 across the 10 of their papers we have counts for
10 papers
MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers
Linrui Ma, Chun Hei Lo, Xinyu Wang +12
The quadratic computational cost of traditional attention mechanisms poses a major bottleneck to the scalability and practical deployment of large language models (LLMs), particula…
Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing
Miao Rang, Zhenni Bi, Hang Zhou +6
Standard knowledge distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, t…
From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
Yuchuan Tian, Yuchen Liang, Shuo Zhang +10
Diffusion Language Models (DLMs) enable fast generation, yet training large DLMs from scratch is costly. As a practical shortcut, adapting off-the-shelf Auto-Regressive (AR) model…
Nexus: Higher-Order Attention Mechanisms in Transformers
Hanting Chen, Chong Zhu, Kai Han +6
Transformers have achieved significant success across various domains, relying on self-attention to capture dependencies. However, the standard first-order attention mechanism is o…
ROOT: Robust Orthogonalized Optimizer for Neural Network Training
Wei He, Kai Han, Hang Zhou +4
The optimization of large language models (LLMs) remains a critical challenge, particularly as model scaling exacerbates sensitivity to algorithmic imprecision and training instabi…
PPE: Positional Preservation Embedding for Token Compression in Multimodal Large Language Models
Mouxiao Huang, Borui Jiang, Dehua Zheng +3
Multimodal large language models (MLLMs) have achieved strong performance on vision-language tasks, yet often suffer from inefficiencies due to redundant visual tokens. Existing to…
OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputs
Yiman Zhang, Ziheng Luo, Qiangyu Yan +4
In this paper, we introduce OmniEval, a benchmark for evaluating omni-modality models like MiniCPM-O 2.6, which encompasses visual, auditory, and textual inputs. Compared with exis…
Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition
Hanting Chen, Yasheng Wang, Kai Han +21
This work presents Pangu Embedded, an efficient Large Language Model (LLM) reasoner developed on Ascend Neural Processing Units (NPUs), featuring flexible fast and slow thinking ca…
Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity
Yehui Tang, Xiaosong Li, Fangcheng Liu +19
The surgence of Mixture of Experts (MoE) in Large Language Models promises a small price of execution cost for a much larger model parameter count and learning capacity, because on…
Physics-Guided Multimodal Transformers are the Necessary Foundation for the Next Generation of Meteorological Science
Jing Han, Hanting Chen, Kai Han +4
This position paper argues that the next generation of artificial intelligence in meteorological and climate sciences must transition from fragmented hybrid heuristics toward a uni…