Mining for meaning: from vision to language through multiple networks consensus

Abstract

Describing visual data into natural language is a very challenging task, at the intersection of computer vision, natural language processing and machine learning. Language goes well beyond the description of physical objects and their interactions and can convey the same abstract idea in many ways. It is both about content at the highest semantic level as well as about fluent form. Here we propose an approach to describe videos in natural language by reaching a consensus among multiple encoder-decoder networks. Finding such a consensual linguistic description, which shares common properties with a larger group, has a better chance to convey the correct meaning. We propose and train several network architectures and use different types of image, audio and video features. Each model produces its own description of the input video and the best one is chosen through an efficient, two-phase consensus process. We demonstrate the strength of our approach by obtaining state of the art results on the challenging MSR-VTT dataset.

arXiv commentAccepted at BMVC 2018

References in corpus (10)

• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• The Kinetics Human Action Video Dataset
• Sequence Level Training with Recurrent Neural Networks
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Sequence to Sequence -- Video to Text
• Describing Videos by Exploiting Temporal Structure
• Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
• Frame- and Segment-Level Features and Candidate Pool Evaluation for Video Caption Generation
• Weakly Supervised Dense Video Captioning
• Video Captioning with Guidance of Multimodal Latent Topics