RePr: Improved Training of Convolutional Filters

Abstract

A well-trained Convolutional Neural Network can easily be pruned without significant loss of performance. This is because of unnecessary overlap in the features captured by the network's filters. Innovations in network architecture such as skip/dense connections and Inception units have mitigated this problem to some extent, but these improvements come with increased computation and memory requirements at run-time. We attempt to address this problem from another angle - not by changing the network structure but by altering the training method. We show that by temporarily pruning and then restoring a subset of the model's filters, and repeating this process cyclically, overlap in the learned features is reduced, producing improved generalization. We show that the existing model-pruning criteria are not optimal for selecting filters to prune in this context and introduce inter-filter orthogonality as the ranking criteria to determine under-expressive filters. Our method is applicable both to vanilla convolutional networks and more complex modern architectures, and improves the performance across a variety of tasks, especially when applied to smaller networks.

arXiv commentCVPR 2019

References in corpus (16)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Neural Architecture Search with Reinforcement Learning
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Feature Pyramid Networks for Object Detection
• Towards Dropout Training for Convolutional Neural Networks
• Provable Bounds for Learning Some Deep Representations
• On the importance of single directions for generalization
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Progressive Neural Architecture Search
• On orthogonality and learning recurrent networks with long term dependencies
• DSD: Dense-Sparse-Dense Training for Deep Neural Networks
• Convergent Learning: Do different neural networks learn the same representations?
• Analyzing noise in autoencoders and deep networks
• NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm

Cited by in corpus (2)

• Gradual Channel Pruning while Training using Feature Relevance Scores for Convolutional Neural Networks
• Accelerating CNN Training by Pruning Activation Gradients