Modularity in Deep Learning: A Survey

Abstract

Modularity is a general principle present in many fields. It offers attractive advantages, including, among others, ease of conceptualization, interpretability, scalability, module combinability, and module reusability. The deep learning community has long sought to take inspiration from the modularity principle, either implicitly or explicitly. This interest has been increasing over recent years. We review the notion of modularity in deep learning around three axes: data, task, and model, which characterize the life cycle of deep learning. Data modularity refers to the observation or creation of data groups for various purposes. Task modularity refers to the decomposition of tasks into sub-tasks. Model modularity means that the architecture of a neural network system can be decomposed into identifiable modules. We describe different instantiations of the modularity principle, and we contextualize their advantages in different deep learning sub-fields. Finally, we conclude the paper with a discussion of the definition of modularity and directions for future research.

References in corpus (25)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Modularity and community structure in networks
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Billion-scale semi-supervised learning for image classification
• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Slimmable Neural Networks
• HyperNetworks
• A Review of Modularization Techniques in Artificial Neural Networks
• Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning
• Routing Networks and the Challenges of Modular and Compositional Computation
• Pathways: Asynchronous Distributed Dataflow for ML
• A Review of Sparse Expert Models in Deep Learning
• Efficient Continual Learning with Modular Networks and Task-Driven Priors
• Comparison and Analysis of Deep Audio Embeddings for Music Emotion Recognition
• Combining Modular Skills in Multitask Learning
• Characterizing Intrinsic Compositionality in Transformers with Tree Projections
• Dynamic Inference with Neural Interpreters
• Modularity in Deep Learning: A Survey
• Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification
• The Role of Global Labels in Few-Shot Classification and How to Infer Them
• Modularity in Reinforcement Learning via Algorithmic Independence in Credit Assignment
• Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning
• Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification

Cited by in corpus (1)

• Modularity in Deep Learning: A Survey