1 citations · 2 across the 30 of their papers we have counts for
17 papers · 1 filter
Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation
Xincheng Wei, Yifan Ding, Yoshua Li +6
On-policy self-distillation (OPSD) trains mathematical reasoning models using a privileged teacher that sees a reference solution and supervises student-sampled prefixes. Standard…
RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges
Kaiyuan Liu, Ziyuan Zhuang, Rongxiang Weng +1
LLM-as-judge is essential for evaluating open-ended text and steering post-training, yet improving the judge itself typically relies on expensive annotations, reward models, or dis…
Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training
Yongwei Zhou, Juncheng Diao, Junlin Shang +2
The efficacy of continued pre-training for Large Language Models (LLMs) hinges upon hyperparameter configurations, such as learning rate and batch size. However, current practices…
Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation
Kaiyuan Liu, Ziyuan Zhuang, Yang Bai +3
On-policy distillation (OPD) trains a student model on its own rollouts using dense feedback from a stronger teacher. Prior literature suggests that, provided teacher feedback is a…
LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance
Yuchun Fan, Bei Li, Peiguang Li +9
Reinforcement learning has proven effective for enhancing multi-step reasoning in large language models (LLMs), yet its benefits have not fully translated to multilingual contexts.…
Unlocking Implicit Experience: Synthesizing Tool-Use Trajectories from Text
Zhihao Xu, Rumei Li, Jiahuan Li +4
Enabling Large Language Models (LLMs) to effectively utilize tools in multi-turn interactions is essential for building capable autonomous agents. However, acquiring diverse and re…
Efficient Context Scaling with LongCat ZigZag Attention
Chen Zhang, Yang Bai, Jiahuan Li +19
We introduce LongCat ZigZag Attention (LoZA), which is a sparse attention scheme designed to transform any existing full-attention models into sparse versions with rather limited c…
A Survey on LLM Mid-Training
Chengying Tu, Xuemiao Zhang, Rongxiang Weng +6
Recent advances in foundation models have highlighted the significant benefits of multi-stage training, with a particular emphasis on the emergence of mid-training as a vital stage…
LongCat-Flash Technical Report
Meituan LongCat Team, Bayan, Bei Li +179
We introduce LongCat-Flash, a 560-billion-parameter Mixture-of-Experts (MoE) language model designed for both computational efficiency and advanced agentic capabilities. Stemming f…
LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points
Xuemiao Zhang, Can Ren, Chengying Tu +4
The advancement of large language models (LLMs) struggles with the scarcity of high-quality, diverse training data. To address this limitation, we propose LinkSyn, a novel knowledg…
Large-Scale Diverse Synthesis for Mid-Training
Xuemiao Zhang, Chengying Tu, Can Ren +4
The scarcity of high-quality, knowledge-intensive training data hinders the development of large language models (LLMs), as traditional corpora provide limited information. Previou…
Libra: Assessing and Improving Reward Model by Learning to Think
Meng Zhou, Bei Li, Jiahao Liu +5
Reinforcement learning (RL) has significantly improved the reasoning ability of large language models. However, current reward models underperform in challenging reasoning scenario…
Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training
Zhijun Wang, Jiahuan Li, Hao Zhou +7
Large language models (LLMs) exhibit remarkable multilingual capabilities despite the extreme language imbalance in the pre-training data. In this paper, we closely examine the rea…
FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategy
Xuemiao Zhang, Feiyu Duan, Liangyu Xu +5
Large language models (LLMs) have significantly advanced human language understanding and generation, with pretraining data quality and organization being crucial to their performa…
FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training
Liangyu Xu, Xuemiao Zhang, Feiyu Duan +4
Selecting high-quality data can improve the pretraining efficiency of large language models (LLMs). Existing methods generally rely on heuristic techniques or single quality signal…
Look Before You Leap: Enhancing Attention and Vigilance Regarding Harmful Content with GuidelineLLM
Shaoqing Zhang, Zhuosheng Zhang, Kehai Chen +4
Despite being empowered with alignment mechanisms, large language models (LLMs) are increasingly vulnerable to emerging jailbreak attacks that can compromise their alignment mechan…
Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
Zhiheng Xi, Dingwen Yang, Jixuan Huang +21
Training large language models (LLMs) to spend more time thinking and reflection before responding is crucial for effectively solving complex reasoning tasks in fields such as scie…