Multi-Task Zipping via Layer-wise Neuron Sharing

Abstract

Future mobile devices are anticipated to perceive, understand and react to the world on their own by running multiple correlated deep neural networks on-device. Yet the complexity of these neural networks needs to be trimmed down both within-model and cross-model to fit in mobile storage and memory. Previous studies focus on squeezing the redundancy within a single neural network. In this work, we aim to reduce the redundancy across multiple models. We propose Multi-Task Zipping (MTZ), a framework to automatically merge correlated, pre-trained deep neural networks for cross-model compression. Central in MTZ is a layer-wise neuron sharing and incoming weight updating scheme that induces a minimal change in the error function. MTZ inherits information from each model and demands light retraining to re-boost the accuracy of individual tasks. Evaluations show that MTZ is able to fully merge the hidden layers of two VGG-16 networks with a 3.18% increase in the test error averaged on ImageNet and CelebA, or share 39.61% parameters between the two networks with <0.5% increase in the test errors for both tasks. The number of iterations to retrain the combined network is at least 17.8 times lower than that of training a single VGG-16 network. Moreover, experiments show that MTZ is also able to effectively merge multiple residual networks.

arXiv commentPublished as a conference paper at NeurIPS 2018

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• How transferable are features in deep neural networks?
• FitNets: Hints for Thin Deep Nets
• Wide Residual Networks
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• A Survey on Multi-Task Learning
• Learning multiple visual domains with residual adapters
• Deep Multi-task Representation Learning: A Tensor Factorisation Approach

Cited by in corpus (5)

• A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions
• Distilling Knowledge from Graph Convolutional Networks
• Feature Partitioning for Efficient Multi-Task Architectures
• Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation
• Pruning-Aware Merging for Efficient Multitask Inference