11 papers
Understanding LLM Quantization through Activation-Guided Compensation and Orthogonal Residuals
Yamato Narita, Issei Sato
Post-training weight-activation quantization reduces the memory and inference costs of large language models, but aggressive W4A4 quantization remains difficult because activation…
Max-pooling Network Revisited: Analyzing the Role of Semantic Probability in Multiple Instance Learning for Hallucination Detection
Shota Fujikawa, Issei Sato
Hallucination detection has become increasingly important for improving the reliability of large language models (LLMs). Recently, hybrid approaches such as HaMI, which combine sem…
Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation
Akiyoshi Tomihari, Issei Sato
Recent analyses question whether reinforcement learning (RL) is responsible for strong reasoning in large language models (LLMs). At the same time, distillation and inference-time…
Understanding Scattered Forest Search: A Version-Space Perspective on Multi-Turn Program Correction
Yuto Tanaka, Issei Sato
In multi-turn program correction, the state-of-the-art method Scattered Forest Search (SFS) has been proposed, employing Monte Carlo Tree Search (MCTS) with carefully crafted initi…
A Formal Comparison Between Chain of Thought and Latent Thought
Kevin Xu, Issei Sato
Chain of thought (CoT) elicits reasoning in large language models by explicitly generating intermediate tokens. In contrast, latent thought reasoning operates directly in the conti…
To CoT or To Loop? A Formal Comparison Between Chain-of-Thought and Looped Transformers
Kevin Xu, Issei Sato
Chain-of-Thought (CoT) and Looped Transformers have been shown to empirically improve performance on reasoning tasks and to theoretically enhance expressivity by recursively increa…
Gradient Heterogeneity Complements Hessian Heterogeneity in Transformer Optimization
Akiyoshi Tomihari, Issei Sato
Transformers are difficult to optimize with stochastic gradient descent (SGD) and largely rely on adaptive optimizers such as Adam. Despite extensive efforts, the mechanisms behind…
Rethinking Associative Memory Mechanism in Induction Head
Shuo Wang, Issei Sato
Induction head mechanism is a part of the computational circuits for in-context learning (ICL) that enable large language models (LLMs) to adapt to new tasks without fine-tuning. M…
On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding
Kevin Xu, Issei Sato
Looped Transformers provide advantages in parameter efficiency, computational capabilities, and generalization for reasoning tasks. However, their expressive power regarding functi…
Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective
Akiyoshi Tomihari, Issei Sato
The two-stage fine-tuning (FT) method, linear probing (LP) then fine-tuning (LP-FT), outperforms linear probing and FT alone. This holds true for both in-distribution (ID) and out-…
Understanding Parameter Saliency via Extreme Value Theory
Shuo Wang, Issei Sato
Deep neural networks are being increasingly implemented throughout society in recent years. It is useful to identify which parameters trigger misclassification in diagnosing undesi…