4 papers
Sumi: Open Uniform Diffusion Language Model from Scratch
Mengyu Ye, Keito Kudo, Wataru Ikeda +3
Diffusion models have become a promising alternative to autoregressive models. Among these, uniform diffusion language models (UDLMs) permit any token to be updated at any step, in…
Suppressing Final Layer Hidden State Jumps in Transformer Pretraining
Keigo Shibata, Kazuki Yano, Ryosuke Takahashi +3
This paper discusses the internal behavior of Transformer language models. Many recent pre-trained models have been reported to exhibit only slight changes in the angular distance…
An Open and Reproducible Deep Research Agent for Long-Form Question Answering
Ikuya Yamada, Wataru Ikeda, Ko Yoshida +5
We present an open deep research system for long-form question answering, selected as a winning system in the text-to-text track of the MMU-RAG competition at NeurIPS 2025. The sys…
Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Models
Wataru Ikeda, Kazuki Yano, Ryosuke Takahashi +3
This study investigates the layerwise importance of feed-forward networks (FFNs) in Transformer-based language models during pretraining. We introduce an experimental approach that…