Lie Access Neural Turing Machine

Abstract

Following the recent trend in explicit neural memory structures, we present a new design of an external memory, wherein memories are stored in an Euclidean key space ${\mathbb{R}}^n$. An LSTM controller performs read and write via specialized read and write heads. It can move a head by either providing a new address in the key space (aka random access) or moving from its previous position via a Lie group action (aka Lie access). In this way, the "L" and "R" instructions of a traditional Turing Machine are generalized to arbitrary elements of a fixed Lie group action. For this reason, we name this new model the Lie Access Neural Turing Machine, or LANTM. We tested two different configurations of LANTM against an LSTM baseline in several basic experiments. We found the right configuration of LANTM to outperform the baseline in all of our experiments. In particular, we trained LANTM on addition of $k$-digit numbers for $2\le k\le 16$, but it was able to generalize almost perfectly to $17\le k\le 32$, all with the number of parameters 2 orders of magnitude below the LSTM baseline.

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Recurrent Neural Network Regularization
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Neural GPUs Learn Algorithms
• Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes
• Speech Recognition with Deep Recurrent Neural Networks

Cited by in corpus (7)

• Learning to Optimize
• Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes
• Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages
• A Survey of Neural Networks and Formal Languages
• Learning Numeracy: Binary Arithmetic with Neural Turing Machines
• Enhancing Reinforcement Learning with discrete interfaces to learn the Dyck Language
• Partially Non-Recurrent Controllers for Memory-Augmented Neural Networks