Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models

Abstract

With an eye toward understanding complexity control in deep learning, we study how infinitesimal regularization or gradient descent optimization lead to margin maximizing solutions in both homogeneous and non-homogeneous models, extending previous work that focused on infinitesimal regularization only in homogeneous models. To this end we study the limit of loss minimization with a diverging norm constraint (the "constrained path"), relate it to the limit of a "margin path" and characterize the resulting solution. For non-homogeneous ensemble models, which output is a sum of homogeneous sub-models, we show that this solution discards the shallowest sub-models if they are unnecessary. For homogeneous models, we show convergence to a "lexicographic max-margin solution", and provide conditions under which max-margin solutions are also attained as the limit of unconstrained gradient descent.

arXiv commentICML Camera ready version

References in corpus (9)

• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Gradient descent aligns the layers of deep linear networks
• Geometry of Optimization and Implicit Regularization in Deep Learning
• Risk and parameter convergence of logistic regression
• Implicit Bias of Gradient Descent on Linear Convolutional Networks
• A Continuous-Time View of Early Stopping for Least Squares
• When Will Gradient Methods Converge to Max-margin Classifier under ReLU Models?

Cited by in corpus (14)

• A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case
• Double descent in the condition number
• Understanding the role of importance weighting for deep learning
• Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks
• Explicit regularization and implicit bias in deep network classifiers trained with the square loss
• Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm
• Theory III: Dynamics and Generalization in Deep Networks
• Implicit bias of deep linear networks in the large learning rate phase
• Distribution of Classification Margins: Are All Data Equal?
• Going Beyond Linear RL: Sample Efficient Neural Function Approximation