Dimension-Free PAC-Bayes Generalization for Deep Neural Networks via Dirichlet Processes

Abstract

Classical PAC-Bayesian generalization theory places prior distributions over Euclidean parameter space $\Theta \subseteq {\mathbb{R}}^D$, causing relative entropy to scale with network dimension $D$. For modern overparameterized neural networks where $D\gg n$, this parameter-counting dependency yields vacuous generalization bounds ($>1,000\%$). In this work, we resolve this barrier by shifting prior specification from internal parameters to the non-parametric space of data distributions ${\mathcal{M}}_1(\mathcal{Z})$ via a \textbf{Dirichlet Process (DP)}, $\mathcal{DP}(\alpha ,F_0)$, inducing parameter distributions through push-forward measures under the learning algorithm. By the Data Processing Inequality, parameter-space relative entropy is strictly bounded by the data-space DP divergence, which we evaluate in exact closed form using Digamma functions. The resulting PAC-Bayesian certificate is strictly invariant to network parameter count $D$, width, and depth. On intrinsic $d^{\ast }$-dimensional submanifolds, optimal partition scaling achieves the minimax non-parametric rate $\mathcal{O}(n^{-1/(d^{\ast }+2)}\sqrt{\ln n})$. Across vision architectures scaling up to 4.2 million parameters ($D=7,850$ to $D=4,203,522$) on the full MNIST benchmark, while classical weight-space bounds scale past $940\%$ (becoming vacuous), our data-space DP-PAC-Bayes certificate remains stable, non-vacuous, and certified at \textbf{$8.8\%$--$9.4\%$}.