11 papers
Efficient RLVR Scheduling via Graph-Structured Online Difficulty Estimation
Zhizhao Liu, Zhiliang Tian, Xi Wang +4
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models but relies on costly rollout exploration. Assigning the same expl…
StateBridge: Training-free Hidden-state Alignment for Latent Communication in LLM Multi-Agent Systems
Yanwen Peng, Delvin Ce Zhang, Xi Wang +1
Large language model based multi-agent systems usually communicate in text, i.e., using discrete tokens. However, text introduces a discrete bottleneck. Converting the sender's con…
Full-bandwidth transformer
Xi Wang, Ziyang Cai, Zheng Zhan +5
Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal acce…
Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinations
Xinyue Fang, Zhiliang Tian, Zhen Huang +5
Existing LLM hallucination mitigation methods, including prompt engineering and model optimization, either hardly alter models'internal knowledge or have poor cross-domain generali…
ATOM-Bench: A Real-World Benchmark for Atomic Skills and Compositional Generalization in Manipulation Policies
Zenan Wu, Bingqing Wei, Lu Liu +8
Generalist manipulation policies are increasingly presented as foundation models for robotic control, but their real-world generalization remains difficult to diagnose. A policy ma…
HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
Mao Lin, Xi Wang, Guilherme Cox +2
As modern LLMs support thousands to millions of tokens, KV caches grow to hundreds of gigabytes, stressing memory capacity and bandwidth. Existing solutions, such as KV cache pruni…
Test-Time Perturbation Learning with Delayed Feedback for Vision-Language-Action Models
Zehua Zang, Xi Wang, Fuchun Sun +4
Vision-Language-Action models (VLAs) achieve remarkable performance in sequential decision-making but remain fragile to subtle environmental shifts, such as small changes in object…
The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise
Xi Wang, Soufiane Hayou, Eric Nalisnick
Mixture of Experts (MoEs) are now ubiquitous in large language models, yet the mechanisms behind their "expert specialization" remain poorly understood. We show that, since MoE rou…
A Tale of Two Temperatures: Simple, Efficient, and Diverse Sampling from Diffusion Language Models
Theo X. Olausson, Metod Jazbec, Xi Wang +4
Much work has been done on designing fast and accurate sampling for diffusion language models (dLLMs). However, these efforts have largely focused on the tradeoff between speed and…
Fundamental Reasoning Paradigms Induce Out-of-Domain Generalization in Language Models
Mingzi Cao, Xingwei Tan, Mahmud Elahi Akhter +4
Deduction, induction, and abduction are fundamental reasoning paradigms, core for human logical thinking. Although improving Large Language Model (LLM) reasoning has attracted sign…
Optimal Transport Depth Up-Scaling
Mingzi Cao, Xi Wang, Nikolaos Aletras
Pre-training Large Language Models (LLMs) from scratch at larger scales yields remarkable performance but incurs substantially high training costs. Depth up-scaling provides an eff…