Understanding and Controlling Memory in Recurrent Neural Networks

Abstract

To be effective in sequential data processing, Recurrent Neural Networks (RNNs) are required to keep track of past events by creating memories. While the relation between memories and the network's hidden state dynamics was established over the last decade, previous works in this direction were of a predominantly descriptive nature focusing mainly on locating the dynamical objects of interest. In particular, it remained unclear how dynamical observables affect the performance, how they form and whether they can be manipulated. Here, we utilize different training protocols, datasets and architectures to obtain a range of networks solving a delayed classification task with similar performance, alongside substantial differences in their ability to extrapolate for longer delays. We analyze the dynamics of the network's hidden state, and uncover the reasons for this difference. Each memory is found to be associated with a nearly steady state of the dynamics which we refer to as a 'slow point'. Slow point speeds predict extrapolation performance across all datasets, protocols and architectures tested. Furthermore, by tracking the formation of the slow points we are able to understand the origin of differences between training protocols. Finally, we propose a novel regularization technique that is based on the relation between hidden state speeds and memory longevity. Our technique manipulates these speeds, thereby leading to a dramatic improvement in memory robustness over time, and could pave the way for a new class of regularization methods.

arXiv commentThe link to the code was changed due to technical issues with the original repository. We no longer refer to the process shown in Figure 5C as a bifurcation diagram, but describe it in a more precise manner. We thank an anonymous reviewer for pointing this out

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• On the difficulty of training Recurrent Neural Networks
• Visualizing and Understanding Recurrent Networks
• End-To-End Memory Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Control of Memory, Active Perception, and Action in Minecraft
• Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networks
• Identifying and Controlling Important Neurons in Neural Machine Translation
• One step back, two steps forward: interference and learning in recurrent neural networks