Modularity in Deep Learning: A Survey
arXiv:2310.01154 · doi:10.1007/978-3-031-37963-5_40
Abstract
Modularity is a general principle present in many fields. It offers attractive advantages, including, among others, ease of conceptualization, interpretability, scalability, module combinability, and module reusability. The deep learning community has long sought to take inspiration from the modularity principle, either implicitly or explicitly. This interest has been increasing over recent years. We review the notion of modularity in deep learning around three axes: data, task, and model, which characterize the life cycle of deep learning. Data modularity refers to the observation or creation of data groups for various purposes. Task modularity refers to the decomposition of tasks into sub-tasks. Model modularity means that the architecture of a neural network system can be decomposed into identifiable modules. We describe different instantiations of the modularity principle, and we contextualize their advantages in different deep learning sub-fields. Finally, we conclude the paper with a discussion of the definition of modularity and directions for future research.
References in corpus (58)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Modularity and community structure in networks
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Prototypical Networks for Few-shot Learning
- An overview of gradient descent optimization algorithms
- PaLM: Scaling Language Modeling with Pathways
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Progressive Neural Networks
- BEiT: BERT Pre-Training of Image Transformers
- PathNet: Evolution Channels Gradient Descent in Super Neural Networks
- Don't Decay the Learning Rate, Increase the Batch Size
- The evolutionary origins of modularity
- Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
- Trunk-Branch Ensemble Convolutional Neural Networks for Video-based Face Recognition
- Billion-scale semi-supervised learning for image classification
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
- Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- Slimmable Neural Networks
- Locating and Editing Factual Associations in GPT
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
- Local modularity measure for network clusterizations
- Exploiting Cyclic Symmetry in Convolutional Neural Networks
- Learning Factored Representations in a Deep Mixture of Experts
- Learning Deep Disentangled Embeddings with the F-Statistic Loss
- HyperNetworks
- Learning Independent Causal Mechanisms
- A Review of Modularization Techniques in Artificial Neural Networks
- Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples
- Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
- Text Recognition in the Wild: A Survey
- Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning
- Routing Networks and the Challenges of Modular and Compositional Computation
- Pathways: Asynchronous Distributed Dataflow for ML
- A Review of Sparse Expert Models in Deep Learning
- Memory-Based Model Editing at Scale
- Efficient Continual Learning with Modular Networks and Task-Driven Priors
- Meta-Learning Symmetries by Reparameterization
- Steerable CNNs
- Comparison and Analysis of Deep Audio Embeddings for Music Emotion Recognition
- Combining Modular Skills in Multitask Learning
- On the Modularity of Hypernetworks
- Is a Modular Architecture Enough?
- Masked Siamese ConvNets
- Characterizing Intrinsic Compositionality in Transformers with Tree Projections
- Dynamic Inference with Neural Interpreters
- Modularity in Deep Learning: A Survey
- Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification
- Network Transplanting
- Modularity in Reinforcement Learning via Algorithmic Independence in Credit Assignment
- The Role of Global Labels in Few-Shot Classification and How to Infer Them
- A Combinatorial Perspective on Transfer Learning
- Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning
- Decomposing Convolutional Neural Networks into Reusable and Replaceable Modules
- Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification
- Network with Sub-Networks