Routing Networks and the Challenges of Modular and Compositional Computation

Abstract

Compositionality is a key strategy for addressing combinatorial complexity and the curse of dimensionality. Recent work has shown that compositional solutions can be learned and offer substantial gains across a variety of domains, including multi-task learning, language modeling, visual question answering, machine comprehension, and others. However, such models present unique challenges during training when both the module parameters and their composition must be learned jointly. In this paper, we identify several of these issues and analyze their underlying causes. Our discussion focuses on routing networks, a general approach to this problem, and examines empirically the interplay of these challenges and a variety of design decisions. In particular, we consider the effect of how the algorithm decides on module composition, how the algorithm updates the modules, and if the algorithm uses regularization.

References in corpus (6)

• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Designing Neural Network Architectures using Reinforcement Learning
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Variational Intrinsic Control
• AdaNet: Adaptive Structural Learning of Artificial Neural Networks
• Selecting Computations: Theory and Applications

Cited by in corpus (7)

• Multi-Task Learning with Deep Neural Networks: A Survey
• Scaling Vision with Sparse Mixture of Experts
• Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives
• Discrete-Valued Neural Communication
• Dynamic Inference with Neural Interpreters
• A Study of Compositional Generalization in Neural Models
• Handling Variable-Dimensional Time Series with Graph Neural Networks