Neural Status Registers
arXiv:2004.07085
Abstract
Standard Neural Networks can learn mathematical operations, but they do not extrapolate. Extrapolation means that the model can apply to larger numbers, well beyond those observed during training. Recent architectures tackle arithmetic operations and can extrapolate; however, the equally important problem of quantitative reasoning remains unaddressed. In this work, we propose a novel architectural element, the Neural Status Register (NSR), for quantitative reasoning over numbers. Our NSR relaxes the discrete bit logic of physical status registers to continuous numbers and allows end-to-end learning with gradient descent. Experiments show that the NSR achieves solutions that extrapolate to numbers many orders of magnitude larger than those in the training set. We successfully train the NSR on number comparisons, piecewise discontinuous functions, counting in sequences, recurrently finding minimums, finding shortest paths in graphs, and comparing digits in images.
References in corpus (12)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Neural Message Passing for Quantum Chemistry
- Measuring abstract reasoning in neural networks
- How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
- Making Neural Programming Architectures Generalize via Recursion
- Neural Arithmetic Logic Units
- Neural Logic Machines
- Improving the Neural GPU Architecture for Algorithm Learning
- Neural Arithmetic Units
- Extrapolation and learning equations
- Measuring Arithmetic Extrapolation Performance
- Neural Arithmetic Expression Calculator