9 papers
LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
Yu Zhao, Zekun Zhang, Fan Jiang +6
Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm. Howeve…
Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
Fan Jiang, Yu Zhao, Chenyang Lyu +5
We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only around 5\% of the total par…
Difficulty-Estimated Policy Optimization
Yu Zhao, Fan Jiang, Tianle Liu +4
Recent advancements in Large Reasoning Models (LRMs), exemplified by DeepSeek-R1, have underscored the potential of scaling inference-time compute through Group Relative Policy Opt…
A State-Transition Framework for Efficient LLM Reasoning
Liang Zhang, Yu Zhao, Longyue Wang +4
While Long Chain-of-Thought (CoT) reasoning significantly improves Large Language Models (LLMs) performance on complex reasoning tasks, the substantial computational and memory cos…
Marco-Bench-MIF: On Multilingual Instruction-Following Capability of Large Language Models
Bo Zeng, Chenyang Lyu, Sinuo Liu +14
Instruction-following capability has become a major ability to be evaluated for Large Language Models (LLMs). However, existing datasets, such as IFEval, are either predominantly m…
Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Models
Huifeng Yin, Yu Zhao, Minghao Wu +9
Large Reasoning Models(LRMs) such as OpenAI o1 and DeepSeek-R1 have shown remarkable reasoning capabilities by scaling test-time compute and generating long Chain-of-Thought(CoT).…