5 papers · 1 filter
Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables
Masaaki Imaizumi, Masanori Koyama, Noboru Isobe +1
We use a mean-field-based transformer model to theoretically investigate how auxiliary variables, such as positional encoding, prevent mode collapse of self-attention mechanisms. T…
Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers
Noboru Isobe, Daisuke Inoue, Masaaki Imaizumi
Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theor…
Flow matching achieves almost minimax optimal convergence
Kenji Fukumizu, Taiji Suzuki, Noboru Isobe +2
Flow matching (FM) has gained significant attention as a simulation-free generative model. Unlike diffusion models, which are based on stochastic differential equations, FM employs…
Extended Flow Matching: a Method of Conditional Generation with Generalized Continuity Equation
Noboru Isobe, Masanori Koyama, Jinzhe Zhang +2
The task of conditional generation is one of the most important applications of generative models, and numerous methods have been developed to date based on the celebrated flow-bas…
A convergence result of a continuous model of deep learning via a Łojasiewicz--Simon inequality
Noboru Isobe
We study an idealized training process for deep neural networks in a continuous-depth, mean-field model in which each layer is parameterized by a probability measure on a Euclidean…