Directional Pruning of Deep Neural Networks

Abstract

In the light of the fact that the stochastic gradient descent (SGD) often finds a flat minimum valley in the training loss, we propose a novel directional pruning method which searches for a sparse minimizer in or close to that flat region. The proposed pruning method does not require retraining or the expert knowledge on the sparsity level. To overcome the computational formidability of estimating the flat directions, we propose to use a carefully tuned ${\ell }_1$ proximal gradient algorithm which can provably achieve the directional pruning with a small learning rate after sufficient training. The empirical results demonstrate the promising results of our solution in highly sparse regime (92% sparsity) among many existing pruning methods on the ResNet50 with the ImageNet, while using only a slightly higher wall time and memory footprint than the SGD. Using the VGG16 and the wide ResNet 28x10 on the CIFAR-10 and CIFAR-100, we demonstrate that our solution reaches the same minima valley as the SGD, and the minima found by our solution and the SGD do not deviate in directions that impact the training loss. The code that reproduces the results of this paper is available at https://github.com/donlan2710/gRDA-Optimizer/tree/master/directional_pruning.

arXiv commentCamera-ready version for the NeurIPS 2020. 29 pages

References in corpus (26)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Reconciling modern machine learning practice and the bias-variance trade-off
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Rethinking the Value of Network Pruning
• The State of Sparsity in Deep Neural Networks
• Variational Dropout Sparsifies Deep Neural Networks
• Essentially No Barriers in Neural Network Energy Landscape
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Stabilizing the Lottery Ticket Hypothesis
• Gradient Descent Happens in a Tiny Subspace
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• When is a Convolutional Filter Easy To Learn?
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• Learning One-hidden-layer ReLU Networks via Gradient Descent
• AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction
• Asymmetric Valleys: Beyond Sharp and Flat Local Minima
• The Difficulty of Training Sparse Neural Networks
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• A generalization of regularized dual averaging and its dynamics
• Tight Sample Complexity of Learning One-hidden-layer Convolutional Neural Networks
• How noise affects the Hessian spectrum in overparameterized neural networks
• Using Mode Connectivity for Loss Landscape Analysis