Routing Networks and the Challenges of Modular and Compositional Computation

Abstract

Compositionality is a key strategy for addressing combinatorial complexity and the curse of dimensionality. Recent work has shown that compositional solutions can be learned and offer substantial gains across a variety of domains, including multi-task learning, language modeling, visual question answering, machine comprehension, and others. However, such models present unique challenges during training when both the module parameters and their composition must be learned jointly. In this paper, we identify several of these issues and analyze their underlying causes. Our discussion focuses on routing networks, a general approach to this problem, and examines empirically the interplay of these challenges and a variety of design decisions. In particular, we consider the effect of how the algorithm decides on module composition, how the algorithm updates the modules, and if the algorithm uses regularization.

References in corpus (13)

• Categorical Reparameterization with Gumbel-Softmax
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Neural Architecture Search via Parameter Sharing
• Designing Neural Network Architectures using Reinforcement Learning
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Variational Intrinsic Control
• Conditional Computation in Neural Networks for faster models
• AdaNet: Adaptive Structural Learning of Artificial Neural Networks
• Automatically Composing Representation Transformations as a Means for Generalization
• Modular Networks: Learning to Decompose Neural Computation
• Selecting Computations: Theory and Applications

Cited by in corpus (19)

• Multi-Task Learning with Deep Neural Networks: A Survey
• Gradient Surgery for Multi-Task Learning
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• Recurrent Independent Mechanisms
• Multi-Task Reinforcement Learning with Soft Modularization
• Scaling Vision with Sparse Mixture of Experts
• Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives
• Coordination Among Neural Modules Through a Shared Global Workspace
• Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systems
• Discrete-Valued Neural Communication
• Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks
• Modularity in Deep Learning: A Survey
• Dynamic Inference with Neural Interpreters
• Flexible Multi-task Networks by Learning Parameter Allocation
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• A Study of Compositional Generalization in Neural Models
• Multi-Head Adapter Routing for Cross-Task Generalization
• Handling Variable-Dimensional Time Series with Graph Neural Networks
• Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Experts