6 papers
Activation-Adjacent Layer Normalization: Effects on Approximation Power
Yunhao Ni, Yuxin Guo, Jie Luo +2
Layer normalization (LN) and root mean square normalization (RMSNorm) depend on their inputs and remain nonlinear at inference. Inserting either next to an activation may therefore…
Kolmogorov-Arnold Networks: Curse of Dimensionality and Structural Bias
Yunhao Ni, Yuxin Guo, Jie Luo +2
Kolmogorov-Arnold networks (KANs) have a dimension-independent approximation exponent when the target admits a sufficiently smooth representation by univariate edge functions. We a…
Separating Intrinsic Ambiguity from Estimation Uncertainty in Deep Generative Models for Linear Inverse Problems
Yuxin Guo, Dongrui Deng, Pulkit Grover
Recently, deep generative models have been used for posterior inference in inverse problems, including high-stakes applications in medical imaging and scientific discovery, where t…
Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm
Yuxin Guo, Yihao Yue, Yunhao Ni +4
Layer normalization (LN) is a fundamental component in modern deep learning, but its per-sample centering and scaling introduce non-negligible inference overhead. RMSNorm improves…
Parallel Layer Normalization for Universal Approximation
Yunhao Ni, Yuxin Guo, Yuhe Liu +4
Layer Normalization (LN) is usually studied for its role in optimization, but it also acts as a multivariate nonlinearity between affine layers. We investigate how grouping LN unit…
On the Nonlinearity of Layer Normalization
Yunhao Ni, Yuxin Guo, Junlong Jia +1
Layer normalization (LN) is a ubiquitous technique in deep learning but our theoretical understanding to it remains elusive. This paper investigates a new theoretical direction for…