6 citations · 8 across the 31 of their papers we have counts for
19 papers · 1 filter
A Preliminary Study on the Promises and Challenges of Native Top- Sparse Attention
Di Xiu, Hongyin Tang, Bolin Rong +4
Large Language Models (LLMs) are increasingly prevalent in the field of long-context modeling, however, their inference computational costs have become a critical bottleneck hinder…
A Survey on LLM Mid-Training
Chengying Tu, Xuemiao Zhang, Rongxiang Weng +6
Recent advances in foundation models have highlighted the significant benefits of multi-stage training, with a particular emphasis on the emergence of mid-training as a vital stage…
Autoformalizer with Tool Feedback
Qi Guo, Jianing Wang, Jianfei Zhang +8
Autoformalization addresses the scarcity of data for Automated Theorem Proving (ATP) by translating mathematical problems from natural language into formal statements. Efforts in r…
OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation
Han Li, Xinyu Peng, Yaoming Wang +7
We introduce OneCAT, a unified multimodal model that seamlessly integrates understanding, generation, and editing within a novel, pure decoder-only transformer architecture. Our fr…
AdaR: A Framework for Equipping LLMs with Adaptive Reasoning
Zhejian Lai, Xiang Geng, Zhijun Wang +7
Mathematical reasoning is a primary indicator of large language models (LLMs) intelligence. However, existing LLMs exhibit failures in robustness and generalization. This paper att…
LongCat-Flash Technical Report
Meituan LongCat Team, Bayan, Bei Li +179
We introduce LongCat-Flash, a 560-billion-parameter Mixture-of-Experts (MoE) language model designed for both computational efficiency and advanced agentic capabilities. Stemming f…
Expanding Reasoning Potential in Foundation Model by Learning Diverse Chains of Thought Patterns
Xuemiao Zhang, Can Ren, Chengying Tu +5
Recent progress in large reasoning models for challenging mathematical reasoning has been driven by reinforcement learning (RL). Incorporating long chain-of-thought (CoT) data duri…
Introducing LongCat-Flash-Thinking: A Technical Report
Meituan LongCat Team, Anchun Gui, Bei Li +122
We present LongCat-Flash-Thinking, an efficient 560-billion-parameter open-source Mixture-of-Experts (MoE) reasoning model. Its advanced capabilities are cultivated through a metic…
LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points
Xuemiao Zhang, Can Ren, Chengying Tu +4
The advancement of large language models (LLMs) struggles with the scarcity of high-quality, diverse training data. To address this limitation, we propose LinkSyn, a novel knowledg…
Large-Scale Diverse Synthesis for Mid-Training
Xuemiao Zhang, Chengying Tu, Can Ren +4
The scarcity of high-quality, knowledge-intensive training data hinders the development of large language models (LLMs), as traditional corpora provide limited information. Previou…
Libra: Assessing and Improving Reward Model by Learning to Think
Meng Zhou, Bei Li, Jiahao Liu +5
Reinforcement learning (RL) has significantly improved the reasoning ability of large language models. However, current reward models underperform in challenging reasoning scenario…
Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang +12
Reward models (RMs) are central to aligning large language models (LLMs) with human values, yet noisy preference feedback can induce reward misgeneralization through spurious corre…
Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment Perspective
Deyang Kong, Qi Guo, Xiangyu Xi +5
Reinforcement learning exhibits potential in enhancing the reasoning abilities of large language models, yet it is hard to scale for the low sample efficiency during the rollout ph…
Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training
Zhijun Wang, Jiahuan Li, Hao Zhou +7
Large language models (LLMs) exhibit remarkable multilingual capabilities despite the extreme language imbalance in the pre-training data. In this paper, we closely examine the rea…
NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables
Lanrui Wang, Mingyu Zheng, Hongyin Tang +5
Processing structured tabular data, particularly large and lengthy tables, constitutes a fundamental yet challenging task for large language models (LLMs). However, existing long-c…
SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversity
Xiangyu Xi, Deyang Kong, Jian Yang +7
Existing pretraining data mixing methods for large language models (LLMs) typically follow a domain-wise methodology, a top-down process that first determines domain weights and th…
Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data
Xuemiao Zhang, Liangyu Xu, Feiyu Duan +5
Large language models (LLMs) generally utilize a consistent data distribution throughout the pretraining process. However, as the model's capability improves, it is intuitive that…
FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategy
Xuemiao Zhang, Feiyu Duan, Liangyu Xu +5
Large language models (LLMs) have significantly advanced human language understanding and generation, with pretraining data quality and organization being crucial to their performa…
FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training
Liangyu Xu, Xuemiao Zhang, Feiyu Duan +4
Selecting high-quality data can improve the pretraining efficiency of large language models (LLMs). Existing methods generally rely on heuristic techniques or single quality signal…