7 citations · 25 across the 29 of their papers we have counts for
34 papers
Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization
Bum Jun Kim
We show that two Transformers representing the same function can follow different training trajectories. For example, doubling a query matrix and halving its key matrix preserves t…
Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation
Bum Jun Kim, Makoto Kawano, Yusuke Iwasawa +1
Modern Transformers rely on many scale-control mechanisms, including normalization, residual scaling, attention temperatures, loss weighting, and adaptive updates. To explain how t…
Detect and Suppress: A Mechanistic Defense against Adversarial Patches in VLA Models
Yukiya Horiba, Koshiro Aoki, Shunsuke Yasuki +2
Adversarial patches can disrupt Vision-Language-Action (VLA) models by manipulating visual observations, leading to failures in robot control. However, it remains poorly understood…
Z-Loss Backward Geometry in Dense Output Heads and Sparse Routers
Bum Jun Kim
Z-loss has been widely applied to the logits of language-model output heads and sparse mixture-of-experts routers. Z-loss constrains the softmax log-normalizers of these output hea…
MiNO: Cotangent-bundle propagator learning for PDEs
Gnankan Landry Regis N'guessan, Bum Jun Kim
Scientific machine learning for partial differential equations commonly targets solution fields, as in physics-informed neural networks, or solution maps, as in neural operators. W…
SEAM: Global consistency beyond local accuracy in scientific machine learning
Gnankan Landry Regis N'guessan, Bum Jun Kim
Scientific machine learning commonly validates models at the level of a subdomain, a benchmark split, or an explanation for one prediction. Yet such local checks cannot establish w…
Looped Transformers with Source-Centered State Evolution
Bum Jun Kim, Kohei Hayashi, Shunsuke Kamiya +3
Looped Transformers create a useful train- and test-time compute axis by reusing the same Transformer block over recurrent depth, increasing effective depth at a fixed parameter co…
Lottery Tickets Are Not Deployment Tickets
Bum Jun Kim
Reports on how sparsification, compression, and lottery tickets change model behavior have been mixed in the prior literature, with beneficial effects observed in some studies and…
SHiPPO: Recurrent Memory with Transported Polynomial Projections
Tomoya Mizuguchi, Bum Jun Kim
HiPPO gives recurrent states memory semantics as coefficients of online polynomial projections, but in fixed channel coordinates. Modern selective SSMs, by contrast, rely on token-…
Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation
Anastasis Kratsios, Simone Brugiapaglia, Bum Jun Kim +2
Feedforward neural network (NN) expressivity is typically studied by emulating optimal basis-expansion schemes. While powerful, this perspective is incomplete: it primarily capture…
EqOD: Symmetry-Informed Stability Selection for PDE Identification
Gnankan Landry Regis N'guessan, Bum Jun Kim
Data-driven identification of partial differential equations (PDEs) relies on sparse regression over a candidate library of differential operators, where larger libraries inflate f…
Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networks
Bum Jun Kim, Gnankan Landry Regis N'guessan
Physics-informed neural networks (PINNs) train a single neural approximation by minimizing multiple physics- and data-derived losses, but the gradients of these losses often interf…
Exactness Matters for Physical Rule Enforcement
Bum Jun Kim
Autoregressive scientific forecasters often enforce physical or structural constraints by repairing each predicted state before feeding it back into the model. However, it remains…
QuadNorm: Resolution-Robust Normalization for Neural Operators
Bum Jun Kim, Makoto Kawano, Yusuke Iwasawa +1
Normalization layers in neural operators usually compute statistics by uniformly averaging discrete grid values, making the normalization itself discretization-dependent and thereb…
Every Feedforward Neural Network Definable in an o-Minimal Structure Has Finite Sample Complexity
Anastasis Kratsios, Gregory Cousins, Haitz Sáez de Ocáriz Borde +2
We show that, in a precise sense, a broad class of feedforward neural networks learn (have finite sample complexity) in the PAC model: every fixed finite feedforward architecture w…
Radial Müntz-Szász Networks: Neural Architectures with Learnable Power Bases for Multidimensional Singularities
Gnankan Landry Regis N'guessan, Bum Jun Kim
Radial singular fields, such as , , and crack-tip profiles, are difficult to model with current coordinate-separable neural architectures. We formally establish this r…
Residual Koopman Spectral Profiling for Predicting and Preventing Transformer Training Instability
Bum Jun Kim, Shohei Taniguchi, Makoto Kawano +2
Training divergence in transformers wastes compute, yet practitioners discover instability only after expensive runs begin. They therefore need an expected probability of failure f…
Discovering Scaling Exponents with Physics-Informed Müntz-Szász Networks
Gnankan Landry Regis N'guessan, Bum Jun Kim
Physical systems near singularities, interfaces, and critical points exhibit power-law scaling, yet standard neural networks leave the governing exponents implicit. We introduce ph…
Unlocking Noise-Resistant Vision: Key Architectural Secrets for Robust Models
Bum Jun Kim, Makoto Kawano, Yusuke Iwasawa +1
While the robustness of vision models is often measured, their dependence on specific architectural design choices is rarely dissected. We investigate why certain vision architectu…
Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connections
Anastasis Kratsios, Bum Jun Kim, Takashi Furuya
Inspired by the Kolmogorov-Arnold superposition theorem, Kolmogorov-Arnold Networks (KANs) have recently emerged as an improved backbone for most deep learning frameworks, promisin…
Temperature-Free Loss Function for Contrastive Learning
Bum Jun Kim, Sang Woo Kim
As one of the most promising methods in self-supervised learning, contrastive learning has achieved a series of breakthroughs across numerous fields. A predominant approach to impl…
Stochastic Subsampling With Average Pooling
Bum Jun Kim, Sang Woo Kim
Regularization of deep neural networks has been an important issue to achieve higher generalization performance without overfitting problems. Although the popular method of Dropout…
The Disappearance of Timestep Embedding in Modern Time-Dependent Neural Networks
Bum Jun Kim, Yoshinobu Kawahara, Sang Woo Kim
Dynamical systems are often time-varying, whose modeling requires a function that evolves with respect to time. Recent studies such as the neural ordinary differential equation pro…
Configuring Data Augmentations to Reduce Variance Shift in Positional Embedding of Vision Transformers
Bum Jun Kim, Sang Woo Kim
Vision transformers (ViTs) have demonstrated remarkable performance in a variety of vision tasks. Despite their promising capabilities, training a ViT requires a large amount of di…