Teaching Machines to Code: Neural Markup Generation with Visual Attention
arXiv:1802.05415
Abstract
We present a neural transducer model with visual attention that learns to generate LaTeX markup of a real-world math formula given its image. Applying sequence modeling and transduction techniques that have been very successful across modalities such as natural language, image, handwriting, speech and audio; we construct an image-to-markup model that learns to produce syntactically and semantically correct LaTeX markup code over 150 words long and achieves a BLEU score of 89%; improving upon the previous state-of-art for the Im2Latex problem. We also demonstrate with heat-map visualization how attention helps in interpreting the model and can pinpoint (detect and localize) symbols on the image accurately despite having been trained without any bounding box data.
arXiv commentFor datasets, visualizations and ancillary material see: https://untrix.github.io/i2l . For source code go to: https://github.com/untrix/im2latex
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Neural Machine Translation by Jointly Learning to Align and Translate
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- WaveNet: A Generative Model for Raw Audio
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Recurrent Neural Network Regularization
- Conditional Image Generation with PixelCNN Decoders
- Pixel Recurrent Neural Networks
- Multi-Scale Context Aggregation by Dilated Convolutions
- Effective Approaches to Attention-based Neural Machine Translation
- Neural Machine Translation in Linear Time
- Listen, Attend and Spell
- How to Construct Deep Recurrent Neural Networks
- Attention Correctness in Neural Image Captioning
- Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph Recognition
- Areas of Attention for Image Captioning