1 citations · 1 across the 7 of their papers we have counts for
6 papers · 1 filter
Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling
Kazuki Yano, Jun Suzuki, Shinji Watanabe
Adapting pre-trained text Large Language Models (LLMs) into Speech Language Models (Speech LMs) via continual pretraining on speech data is promising, but often degrades the origin…
Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning
Kazuki Yano, Shun Kiyono, Sosuke Kobayashi +2
We investigate the role of learning rate scheduling in the large-scale pre-training of large language models, focusing on its influence on downstream performance after supervised f…
Suppressing Final Layer Hidden State Jumps in Transformer Pretraining
Keigo Shibata, Kazuki Yano, Ryosuke Takahashi +3
This paper discusses the internal behavior of Transformer language models. Many recent pre-trained models have been reported to exhibit only slight changes in the angular distance…
Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Models
Wataru Ikeda, Kazuki Yano, Ryosuke Takahashi +3
This study investigates the layerwise importance of feed-forward networks (FFNs) in Transformer-based language models during pretraining. We introduce an experimental approach that…
STEP: Staged Parameter-Efficient Pre-training for Large Language Models
Kazuki Yano, Takumi Ito, Jun Suzuki
Pre-training large language models (LLMs) faces significant memory challenges due to the large size of model parameters. We introduce STaged parameter-Efficient Pre-training (STEP)…
Efficient Construction of Model Family through Progressive Training Using Model Expansion
Kazuki Yano, Sho Takase, Sosuke Kobayashi +2
As Large Language Models (LLMs) gain widespread practical application, offering model families with varying parameter sizes has become standard practice to accommodate diverse comp…