Neural Status Registers
arXiv:2004.07085
Abstract
Standard Neural Networks can learn mathematical operations, but they do not extrapolate. Extrapolation means that the model can apply to larger numbers, well beyond those observed during training. Recent architectures tackle arithmetic operations and can extrapolate; however, the equally important problem of quantitative reasoning remains unaddressed. In this work, we propose a novel architectural element, the Neural Status Register (NSR), for quantitative reasoning over numbers. Our NSR relaxes the discrete bit logic of physical status registers to continuous numbers and allows end-to-end learning with gradient descent. Experiments show that the NSR achieves solutions that extrapolate to numbers many orders of magnitude larger than those in the training set. We successfully train the NSR on number comparisons, piecewise discontinuous functions, counting in sequences, recurrently finding minimums, finding shortest paths in graphs, and comparing digits in images.
References in corpus (16)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Neural Message Passing for Quantum Chemistry
- End-To-End Memory Networks
- Neural GPUs Learn Algorithms
- Measuring abstract reasoning in neural networks
- Learning to Transduce with Unbounded Memory
- How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
- Reinforcement Learning Neural Turing Machines - Revised
- Making Neural Programming Architectures Generalize via Recursion
- Neural Arithmetic Logic Units
- Neural Logic Machines
- Improving the Neural GPU Architecture for Algorithm Learning
- Neural Arithmetic Units
- Extrapolation and learning equations
- Measuring Arithmetic Extrapolation Performance
- Neural Arithmetic Expression Calculator