84 citations · 681 across the 95 of their papers we have counts for
102 papers
Skim and Skip: Hierarchical Adaptive Inference for Efficient Multimodal Retrieval
Meng Gao, Yizhen Zhang, Yang Ding +10
Universal multimodal retrieval (UMR) increasingly adopts multimodal large language models (MLLMs) as unified embedding backbones, but their strong retrieval performance comes at su…
Beyond Static Dialogues: Benchmarking Realistic, Heterogeneous, and Evolving Long-Term Memory
Han Zhang, Zihao Tang, Xin Yu +8
In existing memory benchmarks for Large Language Models (LLMs), the evaluated dialogue sessions often lack long-term semantic consistency, and the underlying personas tend to be fl…
From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents
Murong Ma, Tianyu Chen, Yun Lin +7
Supervised fine-tuning (SFT) on long teacher trajectories is the dominant way to instill investigation and reasoning in open software-engineering (SWE) agents. Since every retained…
Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory
Runxi Cheng, Yuchen Guan, Yongxian Wei +7
Scaling conditional memory offers a promising way to increase language-model capacity, but existing methods such as Engram learn large memory tables from scratch during pre-trainin…
m3BERT: A Modern, Multi-lingual, Matryoshka Bidirectional Encoder
Yaoxiang Wang, Simiao Zuo, Qingguo Hu +4
Embedding models are pivotal in industrial information retrieval systems like search and advertising. However, existing pretrained models often exhibit fixed architectures and embe…
Improving Data and Reward Design for Scientific Reasoning in Large Language Models
Zijie Chen, Zhenghao Lin, Xiao Liu +3
Solving open-ended science questions remains challenging for large language models, particularly due to inherently unreliable supervision and evaluation. The bottleneck lies in the…
Synthesizing File-Level Data for Unit Test Generation with Chain-of-Thoughts via Self-Debugging
Ziyue Hua, Tianyu Chen, Yeyun Gong +8
Automatic unit test (UT) generation is essential for software quality assurance, but existing approaches--including symbolic execution, search-based approaches, and recent LLM-base…
Training LLMs for Divide-and-Conquer Reasoning Elevates Test-Time Scalability
Xiao Liang, Zhong-Zhi Li, Zhenghao Lin +7
Large language models (LLMs) have demonstrated strong reasoning capabilities through step-by-step chain-of-thought (CoT) reasoning. Nevertheless, at the limits of model capability,…
Pull Requests as a Training Signal for Repo-Level Code Editing
Qinglin Zhu, Tianyu Chen, Shuai Lu +8
Repository-level code editing requires models to understand complex dependencies and execute precise multi-file modifications across a large codebase. While recent gains on SWE-ben…
Reducing the Costs of Proof Synthesis on Rust Systems by Scaling Up a Seed Training Set
Nongyu Di, Tianyu Chen, Shan Lu +6
Large Language Models (LLMs) are widely used for code generation. However, the correctness of code generated by LLMs remains a concern. A potential remedy to this concern is to hav…
MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration
Lianhai Ren, Yucheng Ding, Xiao Liu +2
Training instability remains a critical challenge in large language model (LLM) pretraining, often manifesting as sudden gradient explosions that waste significant computational re…
Sigma-MoE-Tiny Technical Report
Qingguo Hu, Zhenghao Lin, Ziyue Yang +12
Mixture-of-Experts (MoE) has emerged as a promising paradigm for foundation models due to its efficient and powerful scalability. In this work, we present Sigma-MoE-Tiny, an MoE la…
SIGMA: An AI-Empowered Training Stack on Early-Life Hardware
Lei Qu, Lianhai Ren, Peng Cheng +12
An increasing variety of AI accelerators is being considered for large-scale training. However, enabling large-scale training on early-life AI accelerators faces three core challen…
Gold-Medal-Level Olympiad Geometry Solving with Efficient Heuristic Auxiliary Constructions
Boyan Duan, Xiao Liang, Shuai Lu +7
Automated theorem proving in Euclidean geometry, particularly for International Mathematical Olympiad (IMO) level problems, remains a major challenge and an important research focu…
Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
Haoran Deng, Yingyu Lin, Zhenghao Lin +4
Long-context language models unlock advanced capabilities in reasoning, code generation, and document summarization by leveraging dependencies across extended spans of text. Howeve…
Learning from the Best, Differently: A Diversity-Driven Rethinking on Data Selection
Hongyi He, Xiao Liu, Zhenghao Lin +6
High-quality pre-training data is crutial for large language models, where quality captures factual reliability and semantic value, and diversity ensures broad coverage and distrib…
Mixture of Neuron Experts
Runxi Cheng, Yuchen Guan, Yucheng Ding +6
In this work, we first explore whether the parameters activated by the MoE layer remain highly sparse at inference. We perform a sparsification study on several representative MoE…
Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
Ruizhe Wang, Yucheng Ding, Xiao Liu +5
As the computational demands for pre-training Large Language Models (LLMs) continue to surge, the need for efficient training paradigms becomes critical. Despite the vast resources…
Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization
Yaoxiang Wang, Qingguo Hu, Yucheng Ding +6
Mixture-of-Experts (MoE) has emerged as a promising paradigm for efficiently scaling large language models without a proportional increase in computational cost. However, the stand…
LayerNorm Induces Recency Bias in Transformer Decoders
Junu Kim, Xiao Liu, Zhenghao Lin +3
Causal self-attention provides positional information to Transformer decoders. Prior work has shown that stacks of causal self-attention layers alone induce a positional bias in at…
Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR
Xiao Liang, Zhongzhi Li, Yeyun Gong +4
Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a key paradigm for post-training Large Language Models (LLMs), particularly for complex reasoning task…
DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections
Haebin Shin, Lei Ji, Xiao Liu +4
As numerous instruction-tuning datasets continue to emerge, dynamically balancing and optimizing their mixtures has become a critical challenge. To address this, we propose Dynamix…
Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
Kailai Yang, Xiao Liu, Lei Ji +6
Continual pre-training on small-scale task-specific data is an effective method for improving large language models in new target fields, yet it risks catastrophic forgetting of th…
Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
Zheng Zhan, Liliang Ren, Shuohang Wang +5
Linear State Space Models (SSMs) offer remarkable performance gains in efficient sequence modeling, with constant inference-time computation and memory complexity. Recent advances,…