Deciding How to Decide: Dynamic Routing in Artificial Neural Networks

Abstract

We propose and systematically evaluate three strategies for training dynamically-routed artificial neural networks: graphs of learned transformations through which different input signals may take different paths. Though some approaches have advantages over others, the resulting networks are often qualitatively similar. We find that, in dynamically-routed networks trained to classify images, layers and branches become specialized to process distinct categories of images. Additionally, given a fixed computational budget, dynamically-routed networks tend to perform better than comparable statically-routed networks.

ICML 2017. Code at https://github.com/MasonMcGill/multipath-nn Video abstract at https://youtu.be/NHQsDaycwyQ

References in corpus (2)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Stacked Hourglass Networks for Human Pose Estimation

Cited by in corpus (18)

• NBDT: Neural-Backed Decision Trees
• Dynamic Neural Networks: A Survey
• AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition
• Switchable Precision Neural Networks
• AdaFrame: Adaptive Frame Selection for Fast Video Recognition
• Anytime Stereo Image Depth Estimation on Mobile Devices
• E${}^2$CM: Early Exit via Class Means for Efficient Supervised and Unsupervised Learning
• AR-Net: Adaptive Frame Resolution for Efficient Action Recognition
• Revisiting Spatial Invariance with Low-Rank Local Connectivity
• Dynamic Compositional Graph Convolutional Network for Efficient Composite Human Motion Prediction
• VA-RED${}^2$: Video Adaptive Redundancy Reduction
• InferLine: ML Prediction Pipeline Provisioning and Management for Tight Latency Objectives
• Dynamic Network Quantization for Efficient Video Inference
• Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers
• Learning Instance-wise Sparsity for Accelerating Deep Models
• Complexity-aware Adaptive Training and Inference for Edge-Cloud Distributed AI Systems
• Channel selection using Gumbel Softmax
• Sparsely ensembled convolutional neural network classifiers via reinforcement learning