Learning to Remember More with Less Memorization
arXiv:1901.01347
Abstract
Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning. Current RAM-like memory models maintain memory accessing every timesteps, thus they do not effectively leverage the short-term memory held in the controller. We hypothesize that this scheme of writing is suboptimal in memory utilization and introduces redundant computation. To validate our hypothesis, we derive a theoretical bound on the amount of information stored in a RAM-like system and formulate an optimization problem that maximizes the bound. The proposed solution dubbed Uniform Writing is proved to be optimal under the assumption of equal timestep contributions. To relax this assumption, we introduce modifications to the original solution, resulting in a solution termed Cached Uniform Writing. This method aims to balance between maximizing memorization and forgetting via overwriting mechanisms. Through an extensive set of experiments, we empirically demonstrate the advantages of our solutions over other recurrent architectures, claiming the state-of-the-arts in various sequential modeling tasks.
Accepted to ICLR'19, oral presentation
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- On the difficulty of training Recurrent Neural Networks
- WaveNet: A Generative Model for Raw Audio
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Adaptive Computation Time for Recurrent Neural Networks
- Generative and Discriminative Text Classification with Recurrent Neural Networks
- Stable Recurrent Models
- Neural Speed Reading via Skim-RNN
- Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes
Cited by in corpus (6)
- Self-Attentive Associative Memory
- CmnRec: Sequential Recommendations with Chunk-accelerated Memory Network
- Meaningful Answer Generation of E-Commerce Question-Answering
- Distributed Associative Memory Network with Memory Refreshing Loss
- Model-Based Episodic Memory Induces Dynamic Hybrid Controls
- Neural Stored-program Memory