1.1k citations · 1.3k across the 94 of their papers we have counts for
24 papers · 1 filter
Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation
Bum Jun Kim, Makoto Kawano, Yusuke Iwasawa +1
Modern Transformers rely on many scale-control mechanisms, including normalization, residual scaling, attention temperatures, loss weighting, and adaptive updates. To explain how t…
Temporally Centered SIGReg Improves LeWorldModel Representations for Robot Policy Learning
Chang Liu, Fei Suo, Yanzhou Jin +4
Recent work on LeWorldModel (LeWM) has shown that the Sketched Isotropic Gaussian Regularizer (SIGReg) enables stable end-to-end world model learning from pixels by regularizing th…
Looped Transformers with Source-Centered State Evolution
Bum Jun Kim, Kohei Hayashi, Shunsuke Kamiya +3
Looped Transformers create a useful train- and test-time compute axis by reusing the same Transformer block over recurrent depth, increasing effective depth at a fixed parameter co…
OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation
Paavo Parmas, Yongmin Kim, Kohsei Matsutani +5
Policy-gradient methods usually optimize expected return, but many real world applications care about distributional properties of returns: tail risk, outlier robustness, or best-o…
On Advantage Estimates for Max@K Policy Gradients
Shota Takashiro, Soichiro Nishimori, Paavo Parmas +6
Reinforcement learning with verifiable rewards is widely used for post-training reasoning models, but sparse outcome rewards make exploration difficult. A complementary approach is…
QuadNorm: Resolution-Robust Normalization for Neural Operators
Bum Jun Kim, Makoto Kawano, Yusuke Iwasawa +1
Normalization layers in neural operators usually compute statistics by uniformly averaging discrete grid values, making the normalization itself discretization-dependent and thereb…
C-voting: Confidence-Based Test-Time Voting without Explicit Energy Functions
Kenji Kubo, Shunsuke Kamiya, Masanori Koyama +3
Neural network models with latent recurrent processing, where identical layers are recursively applied to the latent state, have gained attention as promising models for performing…
Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalization
Shota Takashiro, Masanori Koyama, Takeru Miyato +3
We study out of distribution generalization in streaming tasks where models are trained on short sequences but must operate over much longer, unknown horizons under bounded memory.…
Safe Transformer: An Explicit Safety Bit For Interpretable And Controllable Alignment
Jingyuan Feng, Andrew Gambardella, Gouki Minegishi +3
Current safety alignment methods encode safe behavior implicitly within model parameters, creating a fundamental opacity: we cannot easily inspect why a model refuses a request, no…
Residual Koopman Spectral Profiling for Predicting and Preventing Transformer Training Instability
Bum Jun Kim, Shohei Taniguchi, Makoto Kawano +2
Training divergence in transformers wastes compute, yet practitioners discover instability only after expensive runs begin. They therefore need an expected probability of failure f…
-MoE: Generalizing Mixture of Experts to Infinite Experts
Shota Takashiro, Takeshi Kojima, Shohei Taniguchi +2
The Mixture of Experts (MoE) selects a few feed-forward networks (FFNs) per token, achieving an effective trade-off between computational cost and performance. In conventional MoE,…
Retrieve-Augmented Generation for Speeding up Diffusion Policy without Additional Training
Sodtavilan Odonchimed, Tatsuya Matsushima, Simon Holk +2
Diffusion Policies (DPs) have attracted attention for their ability to achieve significant accuracy improvements in various imitation learning tasks. However, DPs depend on Diffusi…
ADOPT: Modified Adam Can Converge with Any with the Optimal Rate
Shohei Taniguchi, Keno Harada, Gouki Minegishi +7
Adam is one of the most popular optimization algorithms in deep learning. However, it is known that Adam does not converge in theory unless choosing a hyperparameter, i.e., ,…
Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic Tasks
Andrew Gambardella, Yusuke Iwasawa, Yutaka Matsuo
The ability (and inability) of large language models (LLMs) to perform arithmetic tasks has been the subject of much theoretical and practical debate. We show that LLMs are frequen…
Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomials
Hiroki Furuta, Gouki Minegishi, Yusuke Iwasawa +1
Grokking has been actively explored to reveal the mystery of delayed generalization and identifying interpretable representations and algorithms inside the grokked models is a sugg…
Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networks
Gouki Minegishi, Yusuke Iwasawa, Yutaka Matsuo
Grokking is an intriguing phenomenon of delayed generalization, where neural networks initially memorize training data with perfect accuracy but exhibit poor generalization, subseq…
GenORM: Generalizable One-shot Rope Manipulation with Parameter-Aware Policy
So Kuroki, Jiaxian Guo, Tatsuya Matsushima +7
Due to the inherent uncertainty in their deformability during motion, previous methods in rope manipulation often require hundreds of real-world demonstrations to train a manipulat…
End-to-end Training of Deep Boltzmann Machines by Unbiased Contrastive Divergence with Local Mode Initialization
Shohei Taniguchi, Masahiro Suzuki, Yusuke Iwasawa +1
We address the problem of biased gradient estimation in deep Boltzmann machines (DBMs). The existing method to obtain an unbiased estimator uses a maximal coupling based on a Gibbs…
Multimodal Sequential Generative Models for Semi-Supervised Language Instruction Following
Kei Akuzawa, Yusuke Iwasawa, Yutaka Matsuo
Agents that can follow language instructions are expected to be useful in a variety of situations such as navigation. However, training neural network-based agents requires numerou…
A System for Morphology-Task Generalization via Unified Representation and Behavior Distillation
Hiroki Furuta, Yusuke Iwasawa, Yutaka Matsuo +1
The rise of generalist large-scale models in natural language and vision has made us expect that a massive data-driven approach could achieve broader generalization in other domain…
Langevin Autoencoders for Learning Deep Latent Variable Models
Shohei Taniguchi, Yusuke Iwasawa, Wataru Kumagai +1
Markov chain Monte Carlo (MCMC), such as Langevin dynamics, is valid for approximating intractable distributions. However, its usage is limited in the context of deep latent variab…
Group Equivariant Conditional Neural Processes
Makoto Kawano, Wataru Kumagai, Akiyoshi Sannai +2
We present the group equivariant conditional neural process (EquivCNP), a meta-learning method with permutation invariance in a data set as in conventional conditional neural proce…
Wheelchair Behavior Recognition for Visualizing Sidewalk Accessibility by Deep Neural Networks
Takumi Watanabe, Hiroki Takahashi, Goh Sato +3
This paper introduces our methodology to estimate sidewalk accessibilities from wheelchair behavior via a triaxial accelerometer in a smartphone installed under a wheelchair seat.…
Adversarial Invariant Feature Learning with Accuracy Constraint for Domain Generalization
Kei Akuzawa, Yusuke Iwasawa, Yutaka Matsuo
Learning domain-invariant representation is a dominant approach for domain generalization (DG), where we need to build a classifier that is robust toward domain shifts. However, pr…