6 citations · 6 across the 5 of their papers we have counts for
4 papers · 1 filter
Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning
Changhui Sun, Lanbo Liu, Hang Lei +11
On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can…
MDLMPE: Distribution Aware Positional Encoding for Masked Diffusion Language Models
Tong Ling, Hang Lei, Feng Xiao +5
Masked diffusion language models (MDLMs) enable parallel generation and bidirectional context modeling, but their positional context differs fundamentally from that of autoregressi…
Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
Zican Dong, Han Peng, Peiyu Liu +4
Mixture-of-Experts (MoE) models achieve a favorable trade-off between performance and inference efficiency by activating only a subset of experts. However, the memory overhead of s…
Does Conceptual Representation Require Embodiment? Insights From Large Language Models
Qihui Xu, Yingying Peng, Samuel A. Nastase +3
To what extent can language alone give rise to complex concepts, or is embodied experience essential? Recent advancements in large language models (LLMs) offer fresh perspectives o…