Path-SGD: Path-Normalized Optimization in Deep Neural Networks

Abstract

We revisit the choice of SGD for training deep neural networks by reconsidering the appropriate geometry in which to optimize the weights. We argue for a geometry invariant to rescaling of weights that does not affect the output of the network, and suggest Path-SGD, which is an approximate steepest descent method with respect to a path-wise regularizer related to max-norm regularization. Path-SGD is easy and efficient to implement and leads to empirical gains over SGD and AdaGrad.

12 pages, 5 figures

References in corpus (1)

• On the Universality of Online Mirror Descent

Cited by in corpus (85)

• FractalNet: Ultra-Deep Neural Networks without Residuals
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Layer Normalization
• Quantum Natural Gradient
• Exploring Generalization in Deep Learning
• Robust Large Margin Deep Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Generalization in Deep Learning
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Fantastic Generalization Measures and Where to Find Them
• The Role of Neural Network Activation Functions
• Sharp Minima Can Generalize For Deep Nets
• Optimization for deep learning: theory and algorithms
• A Theoretical Analysis of Deep Q-Learning
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• Are All Layers Created Equal?
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Quantum Machine Learning for Particle Physics using a Variational Quantum Classifier
• Stronger generalization bounds for deep nets via a compression approach
• Geometry of Optimization and Implicit Regularization in Deep Learning
• Implicit Regularization in Deep Learning
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Shampoo: Preconditioned Stochastic Tensor Optimization
• A Priori Estimates of the Population Risk for Residual Networks
• What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory
• Banach Space Representer Theorems for Neural Networks and Ridge Splines
• Implicit Bias of Gradient Descent on Linear Convolutional Networks
• Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks
• Deep Frank-Wolfe For Neural Network Optimization
• Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced
• Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks
• A Scale Invariant Flatness Measure for Deep Network Minima
• Normalized Direction-preserving Adam
• Understanding symmetries in deep networks
• Learning Optimal Representations with the Decodable Information Bottleneck
• On Generalization Bounds of a Family of Recurrent Neural Networks
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• On the distance between two neural networks and the stability of learning
• Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
• Riemannian approach to batch normalization
• Iterative Normalization: Beyond Standardization towards Efficient Whitening
• Dropout: Explicit Forms and Capacity Control
• Understanding Generalization in Deep Learning via Tensor Methods
• Symmetry-invariant optimization in deep networks
• Improved Dropout for Shallow and Deep Learning
• Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex
• Unification of Symmetries Inside Neural Networks: Transformer, Feedforward and Neural ODE
• Inductive Bias of Gradient Descent based Adversarial Training on Separable Data
• A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model
• Batch Normalization Preconditioning for Neural Network Training
• Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics
• Perturbated Gradients Updating within Unit Space for Deep Learning
• A Differential Topological View of Challenges in Learning with Feedforward Neural Networks
• Learning compositional functions via multiplicative weight updates
• Function Norms and Regularization in Deep Networks
• Capacity Control of ReLU Neural Networks by Basis-path Norm
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Data-Dependent Path Normalization in Neural Networks
• An Embedding of ReLU Networks and an Analysis of their Identifiability
• Information-Theoretic Local Minima Characterization and Regularization
• $\mathcal{G}$-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
• The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks
• Weighted variation spaces and approximation by shallow ReLU networks
• Extreme Memorization via Scale of Initialization
• Student Specialization in Deep ReLU Networks With Finite Width and Input Dimension
• Practical Assessment of Generalization Performance Robustness for Deep Networks via Contrastive Examples
• Interpreting Basis Path Set in Neural Networks
• The Implicit Bias of AdaGrad on Separable Data
• Effective Regularization Through Loss-Function Metalearning
• A priori generalization error for two-layer ReLU neural network through minimum norm solution
• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
• Orthogonal and Idempotent Transformations for Learning Deep Neural Networks
• On regularization of gradient descent, layer imbalance and flat minima
• Tangent Space Sensitivity and Distribution of Linear Regions in ReLU Networks
• Neural Path Features and Neural Path Kernel : Understanding the role of gates in deep learning
• Characterization of Excess Risk for Locally Strongly Convex Population Risk
• MSR-DARTS: Minimum Stable Rank of Differentiable Architecture Search
• Using Human Psychophysics to Evaluate Generalization in Scene Text Recognition Models
• Stochastic Function Norm Regularization of Deep Networks
• A scale invariant ranking function for learning-to-rank: a real-world use case
• Identifying and Exploiting Structures for Reliable Deep Learning
• Solving hybrid machine learning tasks by traversing weight space geodesics
• Relative Flatness and Generalization