7 papers · 1 filter
Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generation
Ku Onoda, Paavo Parmas, Hiroki Furuta +4
Text-to-image (T2I) models can synthesize realistic, prompt-aligned images, yet samples generated for the same prompt often cover only a small subset of visually distinct modes. Th…
Residual Koopman Spectral Profiling for Predicting and Preventing Transformer Training Instability
Bum Jun Kim, Shohei Taniguchi, Makoto Kawano +2
Training divergence in transformers wastes compute, yet practitioners discover instability only after expensive runs begin. They therefore need an expected probability of failure f…
Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers
Atsushi Shimizu, Shohei Taniguchi, Yutaka Matsuo
Length generalization is the ability of language models to maintain performance on inputs longer than those seen during pretraining. In this work, we introduce a simple yet powerfu…
-MoE: Generalizing Mixture of Experts to Infinite Experts
Shota Takashiro, Takeshi Kojima, Shohei Taniguchi +2
The Mixture of Experts (MoE) selects a few feed-forward networks (FFNs) per token, achieving an effective trade-off between computational cost and performance. In conventional MoE,…
ADOPT: Modified Adam Can Converge with Any with the Optimal Rate
Shohei Taniguchi, Keno Harada, Gouki Minegishi +7
Adam is one of the most popular optimization algorithms in deep learning. However, it is known that Adam does not converge in theory unless choosing a hyperparameter, i.e., ,…
End-to-end Training of Deep Boltzmann Machines by Unbiased Contrastive Divergence with Local Mode Initialization
Shohei Taniguchi, Masahiro Suzuki, Yusuke Iwasawa +1
We address the problem of biased gradient estimation in deep Boltzmann machines (DBMs). The existing method to obtain an unbiased estimator uses a maximal coupling based on a Gibbs…
Langevin Autoencoders for Learning Deep Latent Variable Models
Shohei Taniguchi, Yusuke Iwasawa, Wataru Kumagai +1
Markov chain Monte Carlo (MCMC), such as Langevin dynamics, is valid for approximating intractable distributions. However, its usage is limited in the context of deep latent variab…