Neural Arithmetic Logic Units

Abstract

Neural networks can learn to represent and manipulate numerical information, but they seldom generalize well outside of the range of numerical values encountered during training. To encourage more systematic numerical extrapolation, we propose an architecture that represents numerical quantities as linear activations which are manipulated using primitive arithmetic operators, controlled by learned gates. We call this module a neural arithmetic logic unit (NALU), by analogy to the arithmetic logic unit in traditional processors. Experiments show that NALU-enhanced neural networks can learn to track time, perform arithmetic over images of numbers, translate numerical language into real-valued scalars, execute computer code, and count objects in images. In contrast to conventional architectures, we obtain substantially better generalization both inside and outside of the range of numerical values encountered during training, often extrapolating orders of magnitude beyond trained numerical ranges.

Cited by in corpus (30)

• How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
• Deep Learning for Symbolic Mathematics
• Investigating the Limitations of Transformers with Simple Arithmetic Tasks
• Neural Arithmetic Units
• Pointer Graph Networks
• Measuring Arithmetic Extrapolation Performance
• Learning advanced mathematical computations from examples
• Dangers of Bayesian Model Averaging under Covariate Shift
• Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering
• Neural Logic Rule Layers
• Analogical Reasoning for Visually Grounded Language Acquisition
• Deep learning for pedestrians: backpropagation in CNNs
• Learning the Non-linearity in Convolutional Neural Networks
• A Number Sense as an Emergent Property of the Manipulating Brain
• Neural Status Registers
• Learning to Pool in Graph Neural Networks for Extrapolation
• MC-LSTM: Mass-Conserving LSTM
• On the Transferability of VAE Embeddings using Relational Knowledge with Semi-Supervision
• Factors for the Generalisation of Identity Relations by Neural Networks
• Intelligent Matrix Exponentiation
• Learning Execution through Neural Code Fusion
• Rodent: Relevance determination in differential equations
• A Primer for Neural Arithmetic Logic Modules
• Learning Division with Neural Arithmetic Logic Modules
• Investigating Numeracy Learning Ability of a Text-to-Text Transfer Model
• Pretrained Language Models are Symbolic Mathematics Solvers too!
• Relational Weight Priors in Neural Networks for Abstract Pattern Learning and Language Modelling
• Where's My Head? Definition, Dataset and Models for Numeric Fused-Heads Identification and Resolution
• On the Effectiveness of Low-rank Approximations for Collaborative Filtering compared to Neural Networks
• Truth-Conditional Captioning of Time Series Data