A Primer for Neural Arithmetic Logic Modules

Abstract

Neural Arithmetic Logic Modules have become a growing area of interest, though remain a niche field. These modules are neural networks which aim to achieve systematic generalisation in learning arithmetic and/or logic operations such as $\{+,-,\times ,÷,\le ,\text{AND}\}$ while also being interpretable. This paper is the first in discussing the current state of progress of this field, explaining key works, starting with the Neural Arithmetic Logic Unit (NALU). Focusing on the shortcomings of the NALU, we provide an in-depth analysis to reason about design choices of recent modules. A cross-comparison between modules is made on experiment setups and findings, where we highlight inconsistencies in a fundamental experiment causing the inability to directly compare across papers. To alleviate the existing inconsistencies, we create a benchmark which compares all existing arithmetic NALMs. We finish by providing a novel discussion of existing applications for NALU and research directions requiring further exploration.

JMLR Accepted Version, 58 pages

References in corpus (13)

• Neural Arithmetic Logic Units
• Identity Crisis: Memorization and Generalization under Extreme Overparameterization
• Towards security defect prediction with AI
• Improving the Neural GPU Architecture for Algorithm Learning
• Neural Arithmetic Units
• Neural network gradient-based learning of black-box function interfaces
• Neural Arithmetic Expression Calculator
• Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering
• Analogical Reasoning for Visually Grounded Language Acquisition
• Neural Logic Rule Layers
• Neural Status Registers
• Systematically designing better instance counting models on cell images with Neural Arithmetic Logic Units
• Stealing Black-Box Functionality Using The Deep Neural Tree Architecture

Cited by in corpus (1)

• Learning Division with Neural Arithmetic Logic Modules