A Hybrid Vision Transformer Approach for Mathematical Expression Recognition

Abstract

One of the crucial challenges taken in document analysis is mathematical expression recognition. Unlike text recognition which only focuses on one-dimensional structure images, mathematical expression recognition is a much more complicated problem because of its two-dimensional structure and different symbol size. In this paper, we propose using a Hybrid Vision Transformer (HVT) with 2D positional encoding as the encoder to extract the complex relationship between symbols from the image. A coverage attention decoder is used to better track attention's history to handle the under-parsing and over-parsing problems. We also showed the benefit of using the [CLS] token of ViT as the initial embedding of the decoder. Experiments performed on the IM2LATEX-100K dataset have shown the effectiveness of our method by achieving a BLEU score of 89.94 and outperforming current state-of-the-art methods.

Accepted as oral presentation at DICTA 2022

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Decoupled Weight Decay Regularization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Focusing Attention: Towards Accurate Text Recognition in Natural Images
• Listen, Attend and Spell
• Image-to-Markup Generation with Coarse-to-Fine Attention
• TRIG: Transformer-Based Text Recognizer with Initial Embedding Guidance