5 papers
Mixture of Heterogeneous Grouped Experts for Language Modeling
Zhicheng Ma, Xiang Liu, Zhaoxiang Liu +5
Large Language Models (LLMs) based on Mixture-of-Experts (MoE) are pivotal in industrial applications for their ability to scale performance efficiently. However, standard MoEs enf…
HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation
Wenjing Zhang, Jiangze Yan, Jieyun Huang +7
Distilling reasoning capabilities from Large Reasoning Models (LRMs) into smaller models is typically constrained by the limitation of rejection sampling. Standard methods treat th…
DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Models
Yi Shen, Jian Zhang, Jieyun Huang +7
Recent advancements in slow thinking reasoning models have shown exceptional performance in complex reasoning tasks. However, these models often exhibit overthinking (generating re…
Fuzzy Reasoning Chain (FRC): An Innovative Reasoning Framework from Fuzziness to Clarity
Ping Chen, Xiang Liu, Zhaoxiang Liu +7
With the rapid advancement of large language models (LLMs), natural language processing (NLP) has achieved remarkable progress. Nonetheless, significant challenges remain in handli…
Quantitative Analysis of Performance Drop in DeepSeek Model Quantization
Enbo Zhao, Yi Shen, Shuming Shi +7
Recently, there is a high demand for deploying DeepSeek-R1 and V3 locally, possibly because the official service often suffers from being busy and some organizations have data priv…