2 papers
cs.LG2026
DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization
Tao Zhang, Jianchao Tan, Pingwei Sun +5
Softmax attention stores key and value vectors for every preceding token, causing inference memory to grow with sequence length. Recent language models incorporating Gated DeltaNet…
cs.CL2026
Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning
Changhui Sun, Lanbo Liu, Hang Lei +11
On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can…