Filter Grafting for Deep Neural Networks: Reason, Method, and Cultivation

Abstract

Filter is the key component in modern convolutional neural networks (CNNs). However, since CNNs are usually over-parameterized, a pre-trained network always contain some invalid (unimportant) filters. These filters have relatively small $l_1$ norm and contribute little to the output (\textbf{Reason}). While filter pruning removes these invalid filters for efficiency consideration, we tend to reactivate them to improve the representation capability of CNNs. In this paper, we introduce filter grafting (\textbf{Method}) to achieve this goal. The activation is processed by grafting external information (weights) into invalid filters. To better perform the grafting, we develop a novel criterion to measure the information of filters and an adaptive weighting strategy to balance the grafted information among networks. After the grafting operation, the network has fewer invalid filters compared with its initial state, enpowering the model with more representation capacity. Meanwhile, since grafting is operated reciprocally on all networks involved, we find that grafting may lose the information of valid filters when improving invalid filters. To gain a universal improvement on both valid and invalid filters, we compensate grafting with distillation (\textbf{Cultivation}) to overcome the drawback of grafting . Extensive experiments are performed on the classification and recognition tasks to show the superiority of our method. Code is available at \textcolor{black}{\emph{https://github.com/fxmeng/filter-grafting}}.

arXiv commentarXiv admin note: substantial text overlap with arXiv:2001.05868

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• Opening the Black Box of Deep Neural Networks via Information
• Text Understanding from Scratch
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Data-Free Knowledge Distillation for Deep Neural Networks
• On weight initialization in deep neural networks
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Recurrent Neural Network Training with Dark Knowledge Transfer
• Exploring Linear Relationship in Feature Map Subspace for ConvNets Compression
• SCSP: Spectral Clustering Filter Pruning with Soft Self-adaption Manners