5 papers
Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation
Akiyoshi Tomihari, Issei Sato
Recent analyses question whether reinforcement learning (RL) is responsible for strong reasoning in large language models (LLMs). At the same time, distillation and inference-time…
Gradient Heterogeneity Complements Hessian Heterogeneity in Transformer Optimization
Akiyoshi Tomihari, Issei Sato
Transformers are difficult to optimize with stochastic gradient descent (SGD) and largely rely on adaptive optimizers such as Adam. Despite extensive efforts, the mechanisms behind…
Learning Dynamics in RL Post-Training for Language Models
Akiyoshi Tomihari
Reinforcement learning (RL) post-training is a critical stage in modern language model development, playing a key role in improving alignment and reasoning ability. However, severa…
Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians
Akiyoshi Tomihari, Ryo Karakida
The theoretical understanding of self-attention (SA) has been steadily progressing. A prominent line of work studies a class of SA layers that admit an energy function decreased by…
Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective
Akiyoshi Tomihari, Issei Sato
The two-stage fine-tuning (FT) method, linear probing (LP) then fine-tuning (LP-FT), outperforms linear probing and FT alone. This holds true for both in-distribution (ID) and out-…