Certifying Residual Architectures from Their Primitives: Growth Exponents and Criticality

Abstract

Whether a deep residual architecture avoids numerical blow-up in its forward and backward dynamics is typically determined through costly trial-and-error training. Existing stabilization strategies, including normalization and residual scaling, lack a common framework for predicting their effects across architectures. We develop an a priori certification framework based on two exponents derived from architectural primitives: the forward growth exponent $q$ and the Jacobian sensitivity exponent $\sigma$. These exponents characterize stability regimes and determine how certified allowable depth scales with the floating-point limit $M$. In particular, $q=1$ marks a sharp threshold: for $\sigma >0$, certified depth scales as $\log M/\log \log M$, $\sqrt{\log M}$, and ${\log }_q\log M$ in the regimes $q<1$, $q=1$, and $q>1$, respectively, exhibiting changes reminiscent of phase transitions in statistical physics. The sensitivity exponent $\sigma$ further constrains backward stability, even when $q\le 1$. Exponent arithmetic enables certification from computation graphs, including heterogeneous residual stacks, and identifies targeted modifications to existing backbones without uniform normalization. Experiments on Mamba, PatchTST, FNO, GPT-2, and a Toto~2.0 proxy support the predicted stability distinctions: controlling superlinear growth prevents divergence in Mamba and Toto~2.0, while reducing attention sensitivity restores stable GPT-2 training. Among stable designs, $q=1$ architectures can outperform normalized $q=0$ counterparts in forecasting and operator-learning extrapolation. Rather than prescribing a single architecture, our framework identifies families of architectures with certifiable stability, narrowing the scope of trial-and-error and enabling systematic, task-dependent exploration before training.