6 papers · 1 filter
Finite-Sample Approximation of Hessian-Guided Perturbed Wasserstein Gradient Flows
Ryotaro Kawata, Atsushi Nitanda, Taiji Suzuki
Wasserstein gradient flow extends gradient descent to probability measures. Its Hessian-guided perturbed variant (PWGF) adds Gaussian perturbations to escape saddle points in nonco…
Diffusion Transformers are Provably Optimal In-context Generators
Guoji Fu, Tomoya Wakayama, Ryotaro Kawata +3
Generative foundation models are attracting interest for their ability to produce desired outputs from demonstrations given at inference time, without updating parameters. However,…
Counterfactual Probing for Parallel Unmasking with Hidden Forest Structure
Ryotaro Kawata, Satoshi Hayakawa, Taiji Suzuki
Masked generative models offer parallel token prediction, but accurate parallel sampling must account for dependencies among tokens. When dependencies are unknown, finding safe bat…
From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers
Ryotaro Kawata, Yujin Song, Alberto Bietti +4
Transformers can implement both generalizable algorithms (e.g., induction heads) and simple positional shortcuts (e.g., memorizing fixed output positions). In this work, we study h…
Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning
Ryotaro Kawata, Kohsei Matsutani, Yuri Kinoshita +2
Mixture of Experts (MoE), an ensemble of specialized models equipped with a router that dynamically distributes each input to appropriate experts, has achieved successful results i…
Direct Distributional Optimization for Provable Alignment of Diffusion Models
Ryotaro Kawata, Kazusato Oko, Atsushi Nitanda +1
We introduce a novel alignment method for diffusion models from distribution optimization perspectives while providing rigorous convergence guarantees. We first formulate the probl…