5 papers
Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing
Miao Rang, Zhenni Bi, Hang Zhou +6
Standard knowledge distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, t…
Revealing the Power of Post-Training for Small Language Models via Knowledge Distillation
Miao Rang, Zhenni Bi, Hang Zhou +6
The rapid advancement of large language models (LLMs) has significantly advanced the capabilities of artificial intelligence across various domains. However, their massive scale an…
Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition
Hanting Chen, Yasheng Wang, Kai Han +21
This work presents Pangu Embedded, an efficient Large Language Model (LLM) reasoner developed on Ascend Neural Processing Units (NPUs), featuring flexible fast and slow thinking ca…
Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning
Zhenni Bi, Kai Han, Chuanjian Liu +2
Large Language Models (LLMs) have demonstrated remarkable abilities across various language tasks, but solving complex reasoning problems remains a significant challenge. While exi…
Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts
Miao Rang, Zhenni Bi, Chuanjian Liu +3
Multimodal vision language models (VLMs) have made significant progress with the support of continuously increasing model sizes and data volumes. Running VLMs on edge devices has b…