11 papers
What Keeps Vision-Language Models Looking at the Image?
Hiroto Osaka, Shohei Taniguchi, Gouki Minegishi +3
When do vision-language models need direct access to the image while generating an answer? We study image dependence during answer generation by examining how the visual informatio…
Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generation
Ku Onoda, Paavo Parmas, Hiroki Furuta +4
Text-to-image (T2I) models can synthesize realistic, prompt-aligned images, yet samples generated for the same prompt often cover only a small subset of visually distinct modes. Th…
Residual Koopman Spectral Profiling for Predicting and Preventing Transformer Training Instability
Bum Jun Kim, Shohei Taniguchi, Makoto Kawano +2
Training divergence in transformers wastes compute, yet practitioners discover instability only after expensive runs begin. They therefore need an expected probability of failure f…
Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers
Atsushi Shimizu, Shohei Taniguchi, Yutaka Matsuo
Length generalization is the ability of language models to maintain performance on inputs longer than those seen during pretraining. In this work, we introduce a simple yet powerfu…
-MoE: Generalizing Mixture of Experts to Infinite Experts
Shota Takashiro, Takeshi Kojima, Shohei Taniguchi +2
The Mixture of Experts (MoE) selects a few feed-forward networks (FFNs) per token, achieving an effective trade-off between computational cost and performance. In conventional MoE,…
Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence
Gouki Minegishi, Hiroki Furuta, Shohei Taniguchi +2
Transformer-based language models exhibit In-Context Learning (ICL), where predictions are made adaptively based on context. While prior work links induction heads to ICL through a…
CLIP-like Model as a Foundational Density Ratio Estimator
Fumiya Uchiyama, Rintaro Yanagi, Shohei Taniguchi +5
Density ratio estimation is a core concept in statistical machine learning because it provides a unified mechanism for tasks such as importance weighting, divergence estimation, an…
ADOPT: Modified Adam Can Converge with Any with the Optimal Rate
Shohei Taniguchi, Keno Harada, Gouki Minegishi +7
Adam is one of the most popular optimization algorithms in deep learning. However, it is known that Adam does not converge in theory unless choosing a hyperparameter, i.e., ,…
SSM Meets Video Diffusion Models: Efficient Long-Term Video Generation with Structured State Spaces
Yuta Oshima, Shohei Taniguchi, Masahiro Suzuki +1
Given the remarkable achievements in image generation through diffusion models, the research community has shown increasing interest in extending these models to video generation.…
End-to-end Training of Deep Boltzmann Machines by Unbiased Contrastive Divergence with Local Mode Initialization
Shohei Taniguchi, Masahiro Suzuki, Yusuke Iwasawa +1
We address the problem of biased gradient estimation in deep Boltzmann machines (DBMs). The existing method to obtain an unbiased estimator uses a maximal coupling based on a Gibbs…
Langevin Autoencoders for Learning Deep Latent Variable Models
Shohei Taniguchi, Yusuke Iwasawa, Wataru Kumagai +1
Markov chain Monte Carlo (MCMC), such as Langevin dynamics, is valid for approximating intractable distributions. However, its usage is limited in the context of deep latent variab…