6 papers
UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma
Chongyu Fan, Pengfei Liu, Jingjia Huang +2
Reinforcement learning (RL) has become the standard paradigm for enhancing the complex reasoning capabilities of large language models (LLMs). To achieve sample efficiency, modern…
Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context
Zhaowei Wang, Lishu Luo, Haodong Duan +9
Long-context modeling is becoming a core capability of modern large vision-language models (LVLMs), enabling sustained context management across long-document understanding, video…
MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production
Chunyu Xue, Yangrui Chen, Jianyu Jiang +14
As the foundational component of versatile AI applications, training an multimodal large language model (MLLM) relies on multimodal datasets with dynamic modality mixture proportio…
Mixture-of-Depths Attention
Lianghui Zhu, Yuxin Fang, Bencheng Liao +10
Scaling depth is a key driver for large language models (LLMs). Yet, as LLMs become deeper, they often suffer from signal degradation: informative features formed in shallow layers…
Virtual Width Networks
Seed, Baisheng Li, Banggu Wu +115
We introduce Virtual Width Networks (VWN), a framework that delivers the benefits of wider representations without incurring the quadratic cost of increasing the hidden size. VWN d…
Visual Spatial Tuning
Rui Yang, Ziyu Zhu, Yanwei Li +9
Capturing spatial relationships from visual inputs is a cornerstone of human-like general intelligence. Several previous studies have tried to enhance the spatial awareness of Visi…