7 citations · 25 across the 29 of their papers we have counts for
22 papers · 1 filter
Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization
Bum Jun Kim
We show that two Transformers representing the same function can follow different training trajectories. For example, doubling a query matrix and halving its key matrix preserves t…
Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation
Bum Jun Kim, Makoto Kawano, Yusuke Iwasawa +1
Modern Transformers rely on many scale-control mechanisms, including normalization, residual scaling, attention temperatures, loss weighting, and adaptive updates. To explain how t…
Z-Loss Backward Geometry in Dense Output Heads and Sparse Routers
Bum Jun Kim
Z-loss has been widely applied to the logits of language-model output heads and sparse mixture-of-experts routers. Z-loss constrains the softmax log-normalizers of these output hea…
MiNO: Cotangent-bundle propagator learning for PDEs
Gnankan Landry Regis N'guessan, Bum Jun Kim
Scientific machine learning for partial differential equations commonly targets solution fields, as in physics-informed neural networks, or solution maps, as in neural operators. W…
SEAM: Global consistency beyond local accuracy in scientific machine learning
Gnankan Landry Regis N'guessan, Bum Jun Kim
Scientific machine learning commonly validates models at the level of a subdomain, a benchmark split, or an explanation for one prediction. Yet such local checks cannot establish w…
Looped Transformers with Source-Centered State Evolution
Bum Jun Kim, Kohei Hayashi, Shunsuke Kamiya +3
Looped Transformers create a useful train- and test-time compute axis by reusing the same Transformer block over recurrent depth, increasing effective depth at a fixed parameter co…
Lottery Tickets Are Not Deployment Tickets
Bum Jun Kim
Reports on how sparsification, compression, and lottery tickets change model behavior have been mixed in the prior literature, with beneficial effects observed in some studies and…
SHiPPO: Recurrent Memory with Transported Polynomial Projections
Tomoya Mizuguchi, Bum Jun Kim
HiPPO gives recurrent states memory semantics as coefficients of online polynomial projections, but in fixed channel coordinates. Modern selective SSMs, by contrast, rely on token-…
Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation
Anastasis Kratsios, Simone Brugiapaglia, Bum Jun Kim +2
Feedforward neural network (NN) expressivity is typically studied by emulating optimal basis-expansion schemes. While powerful, this perspective is incomplete: it primarily capture…
EqOD: Symmetry-Informed Stability Selection for PDE Identification
Gnankan Landry Regis N'guessan, Bum Jun Kim
Data-driven identification of partial differential equations (PDEs) relies on sparse regression over a candidate library of differential operators, where larger libraries inflate f…
Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networks
Bum Jun Kim, Gnankan Landry Regis N'guessan
Physics-informed neural networks (PINNs) train a single neural approximation by minimizing multiple physics- and data-derived losses, but the gradients of these losses often interf…
Exactness Matters for Physical Rule Enforcement
Bum Jun Kim
Autoregressive scientific forecasters often enforce physical or structural constraints by repairing each predicted state before feeding it back into the model. However, it remains…
QuadNorm: Resolution-Robust Normalization for Neural Operators
Bum Jun Kim, Makoto Kawano, Yusuke Iwasawa +1
Normalization layers in neural operators usually compute statistics by uniformly averaging discrete grid values, making the normalization itself discretization-dependent and thereb…
Radial Müntz-Szász Networks: Neural Architectures with Learnable Power Bases for Multidimensional Singularities
Gnankan Landry Regis N'guessan, Bum Jun Kim
Radial singular fields, such as , , and crack-tip profiles, are difficult to model with current coordinate-separable neural architectures. We formally establish this r…
Residual Koopman Spectral Profiling for Predicting and Preventing Transformer Training Instability
Bum Jun Kim, Shohei Taniguchi, Makoto Kawano +2
Training divergence in transformers wastes compute, yet practitioners discover instability only after expensive runs begin. They therefore need an expected probability of failure f…
Discovering Scaling Exponents with Physics-Informed Müntz-Szász Networks
Gnankan Landry Regis N'guessan, Bum Jun Kim
Physical systems near singularities, interfaces, and critical points exhibit power-law scaling, yet standard neural networks leave the governing exponents implicit. We introduce ph…
Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connections
Anastasis Kratsios, Bum Jun Kim, Takashi Furuya
Inspired by the Kolmogorov-Arnold superposition theorem, Kolmogorov-Arnold Networks (KANs) have recently emerged as an improved backbone for most deep learning frameworks, promisin…
Temperature-Free Loss Function for Contrastive Learning
Bum Jun Kim, Sang Woo Kim
As one of the most promising methods in self-supervised learning, contrastive learning has achieved a series of breakthroughs across numerous fields. A predominant approach to impl…
Stochastic Subsampling With Average Pooling
Bum Jun Kim, Sang Woo Kim
Regularization of deep neural networks has been an important issue to achieve higher generalization performance without overfitting problems. Although the popular method of Dropout…
The Disappearance of Timestep Embedding in Modern Time-Dependent Neural Networks
Bum Jun Kim, Yoshinobu Kawahara, Sang Woo Kim
Dynamical systems are often time-varying, whose modeling requires a function that evolves with respect to time. Recent studies such as the neural ordinary differential equation pro…
How to Use Dropout Correctly on Residual Networks with Batch Normalization
Bum Jun Kim, Hyeyeon Choi, Hyeonah Jang +2
For the stable optimization of deep neural networks, regularization methods such as dropout and batch normalization have been used in various tasks. Nevertheless, the correct posit…
On the Ideal Number of Groups for Isometric Gradient Propagation
Bum Jun Kim, Hyeyeon Choi, Hyeonah Jang +1
Recently, various normalization layers have been proposed to stabilize the training of deep neural networks. Among them, group normalization is a generalization of layer normalizat…