Using Fast Weights to Attend to the Recent Past

Abstract

Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs. There is no good reason for this restriction. Synapses have dynamics at many different time-scales and this suggests that artificial neural networks might benefit from variables that change slower than activities but much faster than the standard weights. These "fast weights" can be used to store temporary memories of the recent past and they provide a neurally plausible way of implementing the type of attention to the past that has recently proved very helpful in sequence-to-sequence models. By using fast weights we can avoid the need to store copies of neural activity patterns.

Added [Schmidhuber 1993] citation to the last paragraph of the introduction. Fixed typo appendix A.1 uniform initialization to 1/\sqrt{H}

Cited by in corpus (62)

• Meta-Learning for Semi-Supervised Few-Shot Classification
• Deep Reinforcement Learning: An Overview
• Artificial neural networks for neuroscientists: A primer
• Meta-Learning with Latent Embedding Optimization
• Random Feature Attention
• How to train your MAML
• Hopfield Networks is All You Need
• Meta-Learning with Warped Gradient Descent
• Large-Scale Long-Tailed Recognition in an Open World
• On the Binding Problem in Artificial Neural Networks
• Adaptive Siamese Tracking with a Compact Latent Network
• Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution
• Metalearned Neural Memory
• Evolving Modular Soft Robots without Explicit Inter-Module Communication using Local Self-Attention
• Self-Attentive Associative Memory
• HyperGAN: A Generative Model for Diverse, Performant Neural Networks
• Generalisation of structural knowledge in the hippocampal-entorhinal system
• Meta-Learning through Hebbian Plasticity in Random Networks
• Meta-Learning Deep Energy-Based Memory Models
• Fast Parametric Learning with Activation Memorization
• Differentiable plasticity: training plastic neural networks with backpropagation
• Emergent Symbols through Binding in External Memory
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Hard-Attention for Scalable Image Classification
• The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning
• Learning to Perform Role-Filler Binding with Schematic Knowledge
• Learning to update Auto-associative Memory in Recurrent Neural Networks for Improving Sequence Memorization
• Meta Learning Backpropagation And Improving It
• Learning to Learn with Feedback and Local Plasticity
• Finding online neural update rules by learning to remember
• Dynamic Memory Induction Networks for Few-Shot Text Classification
• Meta-Learning Initializations for Image Segmentation
• Few-shot Sequence Learning with Transformers
• Continual Learning with Deep Artificial Neurons
• A Biologically Inspired Visual Working Memory for Deep Networks
• Variational Tracking and Prediction with Generative Disentangled State-Space Models
• Deep Continuous Networks
• Contextual Recurrent Neural Networks
• State-Denoised Recurrent Neural Networks
• Neural Stored-program Memory
• Modeling Past and Future for Neural Machine Translation
• Rotational Unit of Memory
• Learning Associative Inference Using Fast Weight Memory
• Computational models of learning and synaptic plasticity
• Scale-invariant temporal history (SITH): optimal slicing of the past in an uncertain world
• Multirate Training of Neural Networks
• Memory and attention in deep learning
• Deep Learning: Our Miraculous Year 1990-1991
• A Field Guide to Scientific XAI: Transparent and Interpretable Deep Learning for Bioinformatics Research
• Questions to Guide the Future of Artificial Intelligence Research
• A Memory-Augmented Neural Network Model of Abstract Rule Learning
• Zero-shot task adaptation by homoiconic meta-mapping
• Amanuensis: The Programmer's Apprentice
• Fast Weight Long Short-Term Memory
• Beyond Short Clips: End-to-End Video-Level Learning with Collaborative Memories
• Meta-Learning a Dynamical Language Model
• Back to Square One: Superhuman Performance in Chutes and Ladders Through Deep Neural Networks and Tree Search
• One of these (Few) Things is Not Like the Others
• Representation Memorization for Fast Learning New Knowledge without Forgetting
• Dataflow Matrix Machines and V-values: a Bridge between Programs and Neural Nets
• Continual learning under domain transfer with sparse synaptic bursting