Modularized Morphing of Neural Networks

Abstract

In this work we study the problem of network morphism, an effective learning scheme to morph a well-trained neural network to a new one with the network function completely preserved. Different from existing work where basic morphing types on the layer level were addressed, we target at the central problem of network morphism at a higher level, i.e., how a convolutional layer can be morphed into an arbitrary module of a neural network. To simplify the representation of a network, we abstract a module as a graph with blobs as vertices and convolutional layers as edges, based on which the morphing process is able to be formulated as a graph transformation problem. Two atomic morphing operations are introduced to compose the graphs, based on which modules are classified into two families, i.e., simple morphable modules and complex modules. We present practical morphing solutions for both of these two families, and prove that any reasonable module can be morphed from a single convolutional layer. Extensive experiments have been conducted based on the state-of-the-art ResNet on benchmark datasets, and the effectiveness of the proposed solution has been verified.

12 pages, 6 figures, Under review as a conference paper at ICLR 2017

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• Going Deeper with Convolutions

Cited by in corpus (4)

• Forward Thinking: Building and Training Neural Networks One Layer at a Time
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• A Review of Meta-Reinforcement Learning for Deep Neural Networks Architecture Search
• Neural Inheritance Relation Guided One-Shot Layer Assignment Search