2 papers
cs.CL2026
Suppressing Final Layer Hidden State Jumps in Transformer Pretraining
Keigo Shibata, Kazuki Yano, Ryosuke Takahashi +3
This paper discusses the internal behavior of Transformer language models. Many recent pre-trained models have been reported to exhibit only slight changes in the angular distance…
cs.CL2025
Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Models
Wataru Ikeda, Kazuki Yano, Ryosuke Takahashi +3
This study investigates the layerwise importance of feed-forward networks (FFNs) in Transformer-based language models during pretraining. We introduce an experimental approach that…