Associative Long Short-Term Memory

Abstract

We investigate a new method to augment recurrent neural networks with extra memory without increasing the number of network parameters. The system has an associative memory based on complex-valued vectors and is closely related to Holographic Reduced Representations and Long Short-Term Memory networks. Holographic Reduced Representations have limited capacity: as they store more information, each retrieval becomes noisier due to interference. Our system in contrast creates redundant copies of stored information, which enables retrieval with reduced noise. Experiments demonstrate faster learning on multiple memorization tasks.

ICML-2016

References in corpus (3)

• Sequence to Sequence Learning with Neural Networks
• Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
• Neural Turing Machines

Cited by in corpus (33)

• Deep Reinforcement Learning: An Overview
• A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition
• A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations
• A comparison of Vector Symbolic Architectures
• A Survey of Complex-Valued Neural Networks
• Hopfield Networks is All You Need
• Deep Multi-Task Learning with Shared Memory
• Quantum-inspired Complex Convolutional Neural Networks
• Learning to Attend via Word-Aspect Associative Fusion for Aspect-based Sentiment Analysis
• Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies
• Towards Understanding Theoretical Advantages of Complex-Reaction Networks
• Separating Answers from Queries for Neural Reading Comprehension
• Analysis of Deep Complex-Valued Convolutional Neural Networks for MRI Reconstruction
• Exploring Embedding Methods in Binary Hyperdimensional Computing: A Case Study for Motor-Imagery based Brain-Computer Interfaces
• State-Regularized Recurrent Neural Networks
• Deep Quaternion Features for Privacy Protection
• Distributed Associative Memory Network with Memory Refreshing Loss
• Co-VeGAN: Complex-Valued Generative Adversarial Network for Compressive Sensing MR Image Reconstruction
• Representation Learning for Resource Usage Prediction
• Attention Approximates Sparse Distributed Memory
• Interpretable Complex-Valued Neural Networks for Privacy Protection
• Theory of the superposition principle for randomized connectionist representations in neural networks
• BCNN: Binary Complex Neural Network
• Low-pass Recurrent Neural Networks - A memory architecture for longer-term correlation discovery
• Neural Networks with Complex-Valued Weights Have No Spurious Local Minima
• Memory and attention in deep learning
• Recursive Sketches for Modular Deep Learning
• A Novel Framework for Recurrent Neural Networks with Enhancing Information Processing and Transmission between Units
• Quaternion Factorization Machines: A Lightweight Solution to Intricate Feature Interaction Modelling
• Implementation of Optical Deep Neural Networks using the Fabry-Perot Interferometer
• Bayesian Sparsification Methods for Deep Complex-valued Networks
• Learning with Holographic Reduced Representations
• ARMIN: Towards a More Efficient and Light-weight Recurrent Memory Network