Modularized Morphing of Neural Networks

Abstract

In this work we study the problem of network morphism, an effective learning scheme to morph a well-trained neural network to a new one with the network function completely preserved. Different from existing work where basic morphing types on the layer level were addressed, we target at the central problem of network morphism at a higher level, i.e., how a convolutional layer can be morphed into an arbitrary module of a neural network. To simplify the representation of a network, we abstract a module as a graph with blobs as vertices and convolutional layers as edges, based on which the morphing process is able to be formulated as a graph transformation problem. Two atomic morphing operations are introduced to compose the graphs, based on which modules are classified into two families, i.e., simple morphable modules and complex modules. We present practical morphing solutions for both of these two families, and prove that any reasonable module can be morphed from a single convolutional layer. Extensive experiments have been conducted based on the state-of-the-art ResNet on benchmark datasets, and the effectiveness of the proposed solution has been verified.

12 pages, 6 figures, Under review as a conference paper at ICLR 2017

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Deep Residual Learning for Image Recognition
• FitNets: Hints for Thin Deep Nets
• Do Deep Nets Really Need to be Deep?
• Going Deeper with Convolutions
• Identity Mappings in Deep Residual Networks
• Net2Net: Accelerating Learning via Knowledge Transfer

Cited by in corpus (8)

• Neural Architecture Search: A Survey
• AutoML: A Survey of the State-of-the-Art
• Forward Thinking: Building and Training Neural Networks One Layer at a Time
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• A Review of Meta-Reinforcement Learning for Deep Neural Networks Architecture Search
• AutoGrow: Automatic Layer Growing in Deep Convolutional Networks
• On the Transformer Growth for Progressive BERT Training
• Neural Inheritance Relation Guided One-Shot Layer Assignment Search