Differentiable Architecture Pruning for Transfer Learning

Abstract

We propose a new gradient-based approach for extracting sub-architectures from a given large model. Contrarily to existing pruning methods, which are unable to disentangle the network architecture and the corresponding weights, our architecture-pruning scheme produces transferable new structures that can be successfully retrained to solve different tasks. We focus on a transfer-learning setup where architectures can be trained on a large data set but very few data points are available for fine-tuning them on new tasks. We define a new gradient-based algorithm that trains architectures of arbitrarily low complexity independently from the attached weights. Given a search space defined by an existing large neural model, we reformulate the architecture search task as a complexity-penalized subset-selection problem and solve it through a two-temperature relaxation scheme. We provide theoretical convergence guarantees and validate the proposed transfer-learning strategy on real data.

arXiv comment19 pages (main + appendix), 7 figures and 1 table, Workshop @ ICML 2021, 24th July 2021

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Architecture Search with Reinforcement Learning
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• DARTS: Differentiable Architecture Search
• Compressing Neural Networks with the Hashing Trick
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• A Survey on Methods and Theories of Quantized Neural Networks
• Memory Bounded Deep Convolutional Networks
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• From Hashing to CNNs: Training BinaryWeight Networks via Hashing
• Fast Large-Scale Discrete Optimization Based on Principal Coordinate Descent