5 citations · 16 across the 38 of their papers we have counts for
11 papers · 1 filter
Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation
Yaoxiang Wang, Kai Zhang, Ruobing Xie +4
Fine-grained mixture-of-experts (MoE) models commonly increase expert count by narrowing each expert, treating expert shape as a by-product of routing granularity. LatentMoE moves…
Efficient Reasoning Exploration via State-Conditioned Latent Steering with Progress Guidance
Hengyuan Zhang, Chenming Shang, Zunhai Su +10
Best-of- is a widely used inference strategy for complex reasoning, whose effectiveness depends on whether sampled candidates can cover diverse and high-quality reasoning paths.…
Higher-Dimensional Rotary Position Embedding
Yixing Li, Ruobing Xie, Yudong Zhang +3
Transformers rely on position embedding mechanisms in long context modeling in most cases. Rotary Position Embedding (RoPE) embeds positional information with independent 2D rotati…
Hybrid Gated Attention
Zekun Zhou, Ruobing Xie, Lanrui Wang +1
Gated attention is an effective approach to mitigate attention sinks and enhance the representational capacity of attention. To further extend its effectiveness-efficiency Pareto f…
AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
Hong Liu, Rui Cen, Junhan Shi +10
Speculative decoding accelerates large language model inference without changing the target distribution, but no single drafting structure performs best across real-world workloads…
FastMix: Fast Data Mixture Optimization via Gradient Descent
Haoru Tan, Sitong Wu, Yanfeng Chen +5
While large and diverse datasets have driven recent advances in large models, identifying the optimal data mixture for pre-training and post-training remains a significant open pro…
Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs
Wei Ding, Yudong Zhang, Ruobing Xie +3
As foundation models scale toward fusing more heterogeneous visual streams, understanding how diverse encoders interact under joint training becomes a prerequisite for principled d…
Negative Advantages Is a Double-Edged Sword: Calibrating advantages in GRPO for Search Agents
Jiayi Wu, Ruobing Xie, Zeqian Huang +6
Search agents achieve strong question-answering performance through multi-turn interactions with search engines, with Group Relative Policy Optimization (GRPO) being a widely used…
MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs
Wei Ding, Yilin Li, Yudong Zhang +4
Large vision-language models (LVLMs) have achieved remarkable performance across diverse multimodal tasks, yet they continue to suffer from hallucinations, generating content that…
Self-Distillation for Multi-Token Prediction
Guoliang Zhao, Ruobing Xie, An Wang +3
As Large Language Models (LLMs) scale up, inference efficiency becomes a critical bottleneck. Multi-Token Prediction (MTP) could accelerate LLM inference by predicting multiple fut…
Beyond Outliers: A Data-Free Layer-wise Mixed-Precision Quantization Approach Driven by Numerical and Structural Dual-Sensitivity
Hengyuan Zhang, Xinrong Chen, Zunhai Su +10
Layer-wise mixed-precision quantization (LMPQ) enables effective compression under extreme low-bit settings by allocating higher precision to sensitive layers. However, existing me…