End-to-End Attention-based Image Captioning
arXiv:2104.14721
Abstract
In this paper, we address the problem of image captioning specifically for molecular translation where the result would be a predicted chemical notation in InChI format for a given molecular structure. Current approaches mainly follow rule-based or CNN+RNN based methodology. However, they seem to underperform on noisy images and images with small number of distinguishable features. To overcome this, we propose an end-to-end transformer model. When compared to attention-based techniques, our proposed model outperforms on molecular datasets.
References in corpus (9)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Longformer: The Long-Document Transformer
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- Chemception: A Deep Neural Network with Minimal Chemistry Knowledge Matches the Performance of Expert-developed QSAR/QSPR Models
- ChemGrapher: Optical Graph Recognition of Chemical Compounds by Deep Learning
- SMILES2Vec: An Interpretable General-Purpose Deep Neural Network for Predicting Chemical Properties
- CheMixNet: Mixed DNN Architectures for Predicting Chemical Properties using Multiple Molecular Representations