12 papers
On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Zihan Qiu, Zekun Wang, Xiao Li +33
We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and additional 51B parameters of n…
Qwen-CUA: Native Computer Use for (almost) Everything
Dunjie Lu, Shuai Bai, Tianyi Bai +42
Native computer use offers a general interface for agents to operate almost any software available to people, but requires long-horizon state tracking, large-scale interactive expe…
How Benchmarks Mis-Score Computer-Use Agents
Zihan Dong, Zhiyuan Ma, Zekun Wang +5
Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A s…
Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention
Xiao Li, Chengruidong Zhang, Hao Luo +15
Delta-rule linear attention improves recurrent memory updates by correcting what is already stored at the current write address before writing new content. However, the active corr…
Decompose Sparsely Where You Should, Absorb Densely Where You Should No
Ruixuan Deng, Zehao Jin, Zekun Wang +1
Sparse autoencoders (SAEs) are typically trained to reconstruct the \textbf{entire} residual stream through a sparse dictionary, implicitly assuming that all activation content is…
Tool Calling is Linearly Readable and Steerable in Language Models
Zekun Wu, Ze Wang, Seonglae Cho +4
Language-model agents can take real actions by calling tools, so choosing the wrong tool can cause errors that are difficult to undo. Most evaluations only observe the tool choice…
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
Shengkun Tang, Zekun Wang, Bo Zheng +7
Structured pruning and knowledge distillation (KD) are typical techniques for compressing large language models, but it remains unclear how they should be applied at pretraining sc…
Self-Consolidating Language Models: Continual Knowledge Incorporation from Context
Zekun Wang, Anant Gupta, Zihan Dong +1
Large language models (LLMs) increasingly receive information as streams of passages, conversations, and long-context workflows. While longer context windows expose more evidence,…
Accelerating Compound LLM Training Workloads with Maestro
Xiulong Yuan, Hongqing Chen, Jiaxuan Peng +16
Compound LLM training workloads-such as knowledge distillation and multimodal LLM (MLLM) training-are gaining prominence. These typically comprise heterogeneous components differin…
Test-Time Compositional Generalization in Diffusion Models via Concept Discovery
Zekun Wang, Anant Gupta, Tianyi Zhu +1
Compositional generalization requires models to produce novel configurations from familiar parts. In diffusion models, prior compositional generation methods typically assume that…
A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
Zihan Qiu, Zeyu Huang, Kaiyue Wen +16
We investigate the functional role of emergent outliers in large language models, specifically attention sinks (a few tokens that consistently receive large attention logits) and r…
VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos
Dunjie Lu, Yiheng Xu, Junli Wang +12
Training computer-use agents requires massive amounts of GUI interaction data, but manually annotating action trajectories at scale is prohibitively expensive. We present VideoAgen…