Adjoined Networks: A Training Paradigm with Applications to Network Compression

Abstract

Compressing deep neural networks while maintaining accuracy is important when we want to deploy large, powerful models in production and/or edge devices. One common technique used to achieve this goal is knowledge distillation. Typically, the output of a static pre-defined teacher (a large base network) is used as soft labels to train and transfer information to a student (or smaller) network. In this paper, we introduce Adjoined Networks, or AN, a learning paradigm that trains both the original base network and the smaller compressed network together. In our training approach, the parameters of the smaller network are shared across both the base and the compressed networks. Using our training paradigm, we can simultaneously compress (the student network) and regularize (the teacher network) any architecture. In this paper, we focus on popular CNN-based architectures used for computer vision tasks. We conduct an extensive experimental evaluation of our training paradigm on various large-scale datasets. Using ResNet-50 as the base network, AN achieves 71.8% top-1 accuracy with only 1.8M parameters and 1.6 GFLOPs on the ImageNet data-set. We further propose Differentiable Adjoined Networks (DAN), a training paradigm that augments AN by using neural architecture search to jointly learn both the width and the weights for each layer of the smaller network. DAN achieves ResNet-50 level accuracy on ImageNet with $3.8\times$ fewer parameters and $2.2\times$ fewer FLOPs.

Published at AAAI 2022 Spring Symposium on Machine Learning and Knowledge Engineering for Hybrid Intelligence Code available at: https://github.com/utkarshnath/Adjoint-Network.git

References in corpus (13)

• Distilling the Knowledge in a Neural Network
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Neural Architecture Search with Reinforcement Learning
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Large-Scale Evolution of Image Classifiers
• The State of Sparsity in Deep Neural Networks
• Faster CNNs with Direct Sparse Convolutions and Guided Pruning
• Towards Understanding Knowledge Distillation
• Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
• Using Small Proxy Datasets to Accelerate Hyperparameter Search