Norm-Based Capacity Control in Neural Networks

Abstract

We investigate the capacity, convexity and characterization of a general family of norm-constrained feed-forward networks.

29 pages

Cited by in corpus (122)

• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• Benign Overfitting in Linear Regression
• Robust Large Margin Deep Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Generalization in Deep Learning
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?
• The Modern Mathematics of Deep Learning
• A Theoretical Analysis of Deep Q-Learning
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• What Can Neural Networks Reason About?
• The no-free-lunch theorems of supervised learning
• Stronger generalization bounds for deep nets via a compression approach
• Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data
• Layer-adaptive sparsity for the Magnitude-based Pruning
• Analyzing Upper Bounds on Mean Absolute Errors for Deep Neural Network Based Vector-to-Vector Regression
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• Pruning Neural Networks at Initialization: Why are We Missing the Mark?
• On Interpretability of Artificial Neural Networks: A Survey
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• Approximation and Estimation for High-Dimensional Deep Learning Networks
• What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory
• Predicting Neural Network Accuracy from Weights
• Explicit Regularisation in Gaussian Noise Injections
• Lipschitz Continuity in Model-based Reinforcement Learning
• On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning
• A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
• What Do Neural Networks Learn When Trained With Random Labels?
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• Understanding Generalization through Visualizations
• Foundations of Sequence-to-Sequence Modeling for Time Series
• Gradient Descent Quantizes ReLU Network Features
• Reducing Communication for Split Learning by Randomized Top-k Sparsification
• Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks
• How Does Mixup Help With Robustness and Generalization?
• How does Weight Correlation Affect the Generalisation Ability of Deep Neural Networks
• Learning Halfspaces and Neural Networks with Random Initialization
• Learning Optimal Representations with the Decodable Information Bottleneck
• Network size and weights size for memorization with two-layers neural networks
• Effects of quantum resources on the statistical complexity of quantum circuits
• Adversarial Training Versus Weight Decay
• The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers
• Inductive Biases and Variable Creation in Self-Attention Mechanisms
• Exploring Memorization in Adversarial Training
• LocalDrop: A Hybrid Regularization for Deep Neural Networks
• Dropout: Explicit Forms and Capacity Control
• Understanding Generalization in Deep Learning via Tensor Methods
• Generative Modeling with Denoising Auto-Encoders and Langevin Sampling
• How Many Samples are Needed to Estimate a Convolutional or Recurrent Neural Network?
• Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network
• Deep Learning is Singular, and That's Good
• Inductive Bias of Gradient Descent based Adversarial Training on Separable Data
• Stationary Points of Shallow Neural Networks with Quadratic Activation Function
• Deep Neural Networks for Choice Analysis: A Statistical Learning Theory Perspective
• Information-Theoretic Generalization Bounds for Stochastic Gradient Descent
• CNNs Avoid Curse of Dimensionality by Learning on Patches
• On the Depth of Deep Neural Networks: A Theoretical View
• Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformers
• Formalizing Generalization and Robustness of Neural Networks to Weight Perturbations
• Model Complexity of Deep Learning: A Survey
• Distance-Based Regularisation of Deep Networks for Fine-Tuning
• The Role of Information Complexity and Randomization in Representation Learning
• Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error
• Neural Complexity Measures
• Deep Neural Networks for Choice Analysis: Extracting Complete Economic Information for Interpretation
• On generalization bounds for deep networks based on loss surface implicit regularization
• Are deep ResNets provably better than linear predictors?
• Deep Online Convex Optimization with Gated Games
• Function Norms and Regularization in Deep Networks
• Information-Theoretic Local Minima Characterization and Regularization
• Learning with Gradient Descent and Weakly Convex Losses
• Improved Generalization Bounds of Group Invariant / Equivariant Deep Networks via Quotient Feature Spaces
• An Embedding of ReLU Networks and an Analysis of their Identifiability
• Norm-based generalisation bounds for multi-class convolutional neural networks
• In Search of Robust Measures of Generalization
• Data-Dependent Path Normalization in Neural Networks
• De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictors
• A Deep Conditioning Treatment of Neural Networks
• Functional Gradient Boosting based on Residual Network Perception
• Self-Regularity of Non-Negative Output Weights for Overparameterized Two-Layer Neural Networks
• Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Function
• Hyperplane Arrangements of Trained ConvNets Are Biased
• On a Sparse Shortcut Topology of Artificial Neural Networks
• Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations
• Subspace Embeddings Under Nonlinear Transformations
• Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics
• The Rate of Convergence of Variation-Constrained Deep Neural Networks
• On the Robustness and Generalization of Deep Learning Driven Full Waveform Inversion
• An ETF view of Dropout regularization
• Extreme Memorization via Scale of Initialization
• GraphConnect: A Regularization Framework for Neural Networks
• Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum
• Why Mixup Improves the Model Performance
• RATT: Leveraging Unlabeled Data to Guarantee Generalization
• Relative Deviation Margin Bounds
• Stable Rank Normalization for Improved Generalization in Neural Networks and GANs
• Deep Semi-Random Features for Nonlinear Function Approximation
• Bounds for Vector-Valued Function Estimation
• On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications
• Surprising properties of dropout in deep networks
• Why Do Better Loss Functions Lead to Less Transferable Features?
• Fast generalization error bound of deep learning without scale invariance of activation functions
• Revisiting minimum description length complexity in overparameterized models
• Non-linear, Sparse Dimensionality Reduction via Path Lasso Penalized Autoencoders
• The role of invariance in spectral complexity-based generalization bounds
• Uniform Convergence, Adversarial Spheres and a Simple Remedy
• Semantically Robust Unpaired Image Translation for Data with Unmatched Semantics Statistics
• Embedding Inequalities for Barron-type Spaces
• Deep Neural Networks for Choice Analysis: Architectural Design with Alternative-Specific Utility Functions
• Deep Networks as Logical Circuits: Generalization and Interpretation
• A Functional Perspective on Learning Symmetric Functions with Neural Networks
• Information Theoretic Lower Bounds for Feed-Forward Fully-Connected Deep Networks
• Relative Flatness and Generalization
• Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Space
• The duality structure gradient descent algorithm: analysis and applications to neural networks