Describing Multimedia Content using Attention-based Encoder--Decoder Networks

Abstract

Whereas deep neural networks were first mostly used for classification tasks, they are rapidly expanding in the realm of structured output problems, where the observed target is composed of multiple random variables that have a rich joint distribution, given the input. We focus in this paper on the case where the input also has a rich structure and the input and output structures are somehow related. We describe systems that learn to attend to different places in the input, for each element of the output, for a variety of tasks: machine translation, image caption generation, video clip description and speech recognition. All these systems are based on a shared set of building blocks: gated recurrent neural networks and convolutional neural networks, along with trained attention mechanisms. We report on experimental results with these systems, showing impressively good performance and the advantage of the attention mechanism.

Submitted to IEEE Transactions on Multimedia Special Issue on Deep Learning for Multimedia Computing

Cited by in corpus (15)

• Predicting Cardiovascular Risk Factors from Retinal Fundus Photographs using Deep Learning
• Recent advances and clinical applications of deep learning in medical image analysis
• Deep attractor network for single-microphone speaker separation
• Predicting Visual Features from Text for Image and Video Caption Retrieval
• Combining time-series and textual data for taxi demand prediction in event areas: a deep learning approach
• Mining Fashion Outfit Composition Using An End-to-End Deep Learning Approach on Set Data
• Pay attention to the activations: a modular attention mechanism for fine-grained image recognition
• Infrared and 3D skeleton feature fusion for RGB-D action recognition
• Neural Attention for Image Captioning: Review of Outstanding Methods
• Beyond saliency: understanding convolutional neural networks from saliency prediction on layer-wise relevance propagation
• COMIC: Towards A Compact Image Captioning Model with Attention
• Recommender systems based on graph embedding techniques: A comprehensive review
• Character-Based Handwritten Text Transcription with Attention Networks
• LadRa-Net: Locally-Aware Dynamic Re-read Attention Net for Sentence Semantic Matching
• Neural Twins Talk & Alternative Calculations