A Review of Modularization Techniques in Artificial Neural Networks

Abstract

Artificial neural networks (ANNs) have achieved significant success in tackling classical and modern machine learning problems. As learning problems grow in scale and complexity, and expand into multi-disciplinary territory, a more modular approach for scaling ANNs will be needed. Modular neural networks (MNNs) are neural networks that embody the concepts and principles of modularity. MNNs adopt a large number of different techniques for achieving modularization. Previous surveys of modularization techniques are relatively scarce in their systematic analysis of MNNs, focusing mostly on empirical comparisons and lacking an extensive taxonomical framework. In this review, we aim to establish a solid taxonomy that captures the essential properties and relationships of the different variants of MNNs. Based on an investigation of the different levels at which modularization techniques act, we attempt to provide a universal and systematic framework for theorists studying MNNs, also trying along the way to emphasise the strengths and weaknesses of different modularization approaches in order to highlight good practices for neural network practitioners.

arXiv commentArtif Intell Rev (2019)

References in corpus (6)

• Modularity and community structure in networks
• Defining and identifying communities in networks
• The evolutionary origins of modularity
• Community detection in networks: Modularity optimization and maximum likelihood are equivalent
• Noisy matrix decomposition via convex relaxation: Optimal rates in high dimensions
• Improving Neural Network Generalization by Combining Parallel Circuits with Dropout

Cited by in corpus (5)

• A Survey of the Recent Architectures of Deep Convolutional Neural Networks
• Regularized Deep Networks in Intelligent Transportation Systems: A Taxonomy and a Case Study
• Applications of Spiking Neural Networks in Visual Place Recognition
• Modularity in Deep Learning: A Survey
• Multi-Excitation Projective Simulation with a Many-Body Physics Inspired Inductive Bias