Contrasting random and learned features in deep Bayesian linear regression

Abstract

Understanding how feature learning affects generalization is among the foremost goals of modern deep learning theory. Here, we study how the ability to learn representations affects the generalization performance of a simple class of models: deep Bayesian linear neural networks trained on unstructured Gaussian data. By comparing deep random feature models to deep networks in which all layers are trained, we provide a detailed characterization of the interplay between width, depth, data density, and prior mismatch. We show that both models display sample-wise double-descent behavior in the presence of label noise. Random feature models can also display model-wise double-descent if there are narrow bottleneck layers, while deep networks do not show these divergences. Random feature models can have particular widths that are optimal for generalization at a given data density, while making neural networks as wide or as narrow as possible is always optimal. Moreover, we show that the leading-order correction to the kernel-limit learning curve cannot distinguish between random feature models and deep networks in which all layers are trained. Taken together, our findings begin to elucidate how architectural details affect generalization performance in this simple class of deep regression models.

arXiv comment35 pages, 7 figures. v2: minor typos corrected and references added; published in PRE

References in corpus (16)

• SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
• Array Programming with NumPy
• Reconciling modern machine learning practice and the bias-variance trade-off
• The Principles of Deep Learning Theory
• Products of Rectangular Random Matrices: Singular Values and Progressive Scattering
• Scaling description of generalization with number of parameters in deep learning
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
• Weak Commutation Relations and Eigenvalue Statistics for Products of Rectangular Random Matrices
• Neural Networks and Quantum Field Theory
• Disentangling feature and lazy training in deep neural networks
• Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization
• Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm
• Statistical Mechanics of High-Dimensional Inference
• Activation function dependence of the storage capacity of treelike neural networks
• Phase transitions in soft-committee machines

Cited by in corpus (3)

• A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
• Asymptotics of representation learning in finite Bayesian neural networks
• Unified field theoretical approach to deep and recurrent neuronal networks