Showing 2025 · cs.CLShow all
2 papers · 2 filters
cs.CL2025
LayerNorm Induces Recency Bias in Transformer Decoders
Junu Kim, Xiao Liu, Zhenghao Lin +3
Causal self-attention provides positional information to Transformer decoders. Prior work has shown that stacks of causal self-attention layers alone induce a positional bias in at…
cs.CL2025
Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling
Haebin Shin, Lei Ji, Xiao Liu +1
Using large teacher models to guide the training of smaller student models has become the prevailing paradigm for efficient and effective learning. However, vocabulary mismatches b…