Modular Networks: Learning to Decompose Neural Computation

Abstract

Scaling model capacity has been vital in the success of deep learning. For a typical network, necessary compute resources and training time grow dramatically with model size. Conditional computation is a promising way to increase the number of parameters with a relatively small increase in resources. We propose a training algorithm that flexibly chooses neural modules based on the data to be processed. Both the decomposition and modules are learned end-to-end. In contrast to existing approaches, training does not rely on regularization to enforce diversity in module use. We apply modular networks both to image recognition and language modeling tasks, where we achieve superior performance compared to several baselines. Introspection reveals that modules specialize in interpretable contexts.

NIPS 2018

Cited by in corpus (16)

• Multi-Task Learning with Deep Neural Networks: A Survey
• Recurrent Independent Mechanisms
• Routing Networks and the Challenges of Modular and Compositional Computation
• Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systems
• Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks
• Lifelong Learning of Compositional Structures
• Dynamic Inference with Neural Interpreters
• Meta Learning Backpropagation And Improving It
• Clusterability in Neural Networks
• Compositional Models: Multi-Task Learning and Knowledge Transfer with Modular Networks
• Question Guided Modular Routing Networks for Visual Question Answering
• Towards Modular Algorithm Induction
• Dynamic Routing Networks
• Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Experts
• A modularity comparison of Long Short-Term Memory and Morphognosis neural networks
• Understanding the Dynamics of DNNs Using Graph Modularity