1 citations · 2 across the 18 of their papers we have counts for
4 papers · 1 filter
What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillation
Kanghui Tian, Siyuan Liu, Tianxiang Jiang +8
More privileged information does not always make a better teacher. We study this tension in on-policy self-distillation (OPSD), where a self-teacher scores the student's own rollou…
How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization
Xinyi Wu, Siyuan Liu, Ali Jadbabaie
Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly. We study what det…
Data Difficulty and the Generalization--Extrapolation Tradeoff in LLM Fine-Tuning
Siyuan Liu, Tinghong Chen, Xinghan Li +2
Data selection during supervised fine-tuning (SFT) can critically change the behavior of large language models (LLMs). Although existing work has studied the effect of selecting da…
Two-Stage Regularization-Based Structured Pruning for LLMs
Mingkuan Feng, Jinyang Wu, Siyuan Liu +7
The deployment of large language models (LLMs) is largely hindered by their large number of parameters. Structural pruning has emerged as a promising solution. Prior structured pru…