Visually Grounded Word Embeddings and Richer Visual Features for Improving Multimodal Neural Machine Translation
arXiv:1707.01009 · doi:10.21437/GLU.2017-13
Abstract
In Multimodal Neural Machine Translation (MNMT), a neural model generates a translated sentence that describes an image, given the image itself and one source descriptions in English. This is considered as the multimodal image caption translation task. The images are processed with Convolutional Neural Network (CNN) to extract visual features exploitable by the translation model. So far, the CNNs used are pre-trained on object detection and localization task. We hypothesize that richer architecture, such as dense captioning models, may be more suitable for MNMT and could lead to improved translations. We extend this intuition to the word-embeddings, where we compute both linguistic and visual representation for our corpus vocabulary. We combine and compare different confi
arXiv commentAccepted to GLU 2017. arXiv admin note: text overlap with arXiv:1707.00995
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Neural Machine Translation by Jointly Learning to Align and Translate
- Sequence to Sequence Learning with Neural Networks
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- ADADELTA: An Adaptive Learning Rate Method
- A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
- Grammar as a Foreign Language
- Imagination improves Multimodal Translation
- Does Multimodality Help Human and Machine for Translation and Image Captioning?
- An empirical study on the effectiveness of images in Multimodal Neural Machine Translation
- Multimodal Compact Bilinear Pooling for Multimodal Neural Machine Translation