Attending to Mathematical Language with Transformers

Abstract

Mathematical expressions were generated, evaluated and used to train neural network models based on the transformer architecture. The expressions and their targets were analyzed as a character-level sequence transduction task in which the encoder and decoder are built on attention mechanisms. Three models were trained to understand and evaluate symbolic variables and expressions in mathematics: (1) the self-attentive and feed-forward transformer without recurrence or convolution, (2) the universal transformer with recurrence, and (3) the adaptive universal transformer with recurrence and adaptive computation time. The models respectively achieved test accuracies as high as 76.1%, 78.8% and 84.9% in evaluating the expressions to match the target values. For the cases inferred incorrectly, the results differed from the targets by only one or two characters. The models notably learned to add, subtract and multiply both positive and negative decimal numbers of variable digits assigned to symbolic variables.

References in corpus (11)

• Adaptive Computation Time for Recurrent Neural Networks
• Learning to Execute
• Grid Long Short-Term Memory
• Neural GPUs Learn Algorithms
• Learning to Discover Efficient Mathematical Identities
• Improving the Neural GPU Architecture for Algorithm Learning
• Churn analysis using deep convolutional neural networks and autoencoders
• Visual Learning of Arithmetic Operations
• Extensions and Limitations of the Neural GPU
• Comparing heterogeneous entities using artificial neural networks of trainable weighted structural components and machine-learned activation functions
• A Collaborative Approach to Angel and Venture Capital Investment Recommendations

Cited by in corpus (8)

• On Identifiability in Transformers
• Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
• Multi-Label Product Categorization Using Multi-Modal Fusion Models
• Question Generation by Transformers
• Transfer Learning Robustness in Multi-Class Categorization by Fine-Tuning Pre-Trained Contextualized Language Models
• Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoning
• FastTrees: Parallel Latent Tree-Induction for Faster Sequence Encoding