Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers
arXiv:2010.08012
Abstract
Feed-forward neural networks consist of a sequence of layers, in which each layer performs some processing on the information from the previous layer. A downside to this approach is that each layer (or module, as multiple modules can operate in parallel) is tasked with processing the entire hidden state, rather than a particular part of the state which is most relevant for that module. Methods which only operate on a small number of input variables are an essential part of most programming languages, and they allow for improved modularity and code re-usability. Our proposed method, Neural Function Modules (NFM), aims to introduce the same structural capability into deep learning. Most of the work in the context of feed-forward networks combining top-down and bottom-up feedback is limited to classification problems. The key contribution of our work is to combine attention, sparsity, top-down and bottom-up feedback, in a flexible algorithm which, as we show, improves the results in standard classification, out-of-domain generalization, generative modeling, and learning representations in the context of reinforcement learning.
References in corpus (21)
- mixup: Beyond Empirical Risk Minimization
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Spectral Normalization for Generative Adversarial Networks
- Improved Regularization of Convolutional Neural Networks with Cutout
- Self-Attention Generative Adversarial Networks
- Improved Techniques for Training GANs
- PathNet: Evolution Channels Gradient Descent in Super Neural Networks
- Neural Ordinary Differential Equations
- Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks
- Generating Long Sequences with Sparse Transformers
- Manifold Mixup: Better Representations by Interpolating Hidden States
- Implicit Quantile Networks for Distributional Reinforcement Learning
- A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
- Residual Connections Encourage Iterative Inference
- Recurrent Independent Mechanisms
- Deciding How to Decide: Dynamic Routing in Artificial Neural Networks
- LOGAN: Latent Optimisation for Generative Adversarial Networks
- Iterative Amortized Inference
- Systematic Generalization: What Is Required and Can It Be Learned?
- Mimicry: Towards the Reproducibility of GAN Research
- Dilated DenseNets for Relational Reasoning