7 papers · 1 filter
Understanding LLM Quantization through Activation-Guided Compensation and Orthogonal Residuals
Yamato Narita, Issei Sato
Post-training weight-activation quantization reduces the memory and inference costs of large language models, but aggressive W4A4 quantization remains difficult because activation…
Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation
Akiyoshi Tomihari, Issei Sato
Recent analyses question whether reinforcement learning (RL) is responsible for strong reasoning in large language models (LLMs). At the same time, distillation and inference-time…
Understanding Scattered Forest Search: A Version-Space Perspective on Multi-Turn Program Correction
Yuto Tanaka, Issei Sato
In multi-turn program correction, the state-of-the-art method Scattered Forest Search (SFS) has been proposed, employing Monte Carlo Tree Search (MCTS) with carefully crafted initi…
To CoT or To Loop? A Formal Comparison Between Chain-of-Thought and Looped Transformers
Kevin Xu, Issei Sato
Chain-of-Thought (CoT) and Looped Transformers have been shown to empirically improve performance on reasoning tasks and to theoretically enhance expressivity by recursively increa…
Gradient Heterogeneity Complements Hessian Heterogeneity in Transformer Optimization
Akiyoshi Tomihari, Issei Sato
Transformers are difficult to optimize with stochastic gradient descent (SGD) and largely rely on adaptive optimizers such as Adam. Despite extensive efforts, the mechanisms behind…
On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding
Kevin Xu, Issei Sato
Looped Transformers provide advantages in parameter efficiency, computational capabilities, and generalization for reasoning tasks. However, their expressive power regarding functi…
Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective
Akiyoshi Tomihari, Issei Sato
The two-stage fine-tuning (FT) method, linear probing (LP) then fine-tuning (LP-FT), outperforms linear probing and FT alone. This holds true for both in-distribution (ID) and out-…