Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods

Abstract

Interest in Artificial Intelligence (AI) and its applications has seen unprecedented growth in the last few years. This success can be partly attributed to the advancements made in the sub-fields of AI such as machine learning, computer vision, and natural language processing. Much of the growth in these fields has been made possible with deep learning, a sub-area of machine learning that uses artificial neural networks. This has created significant interest in the integration of vision and language. In this survey, we focus on ten prominent tasks that integrate language and vision by discussing their problem formulation, methods, existing datasets, evaluation measures, and compare the results obtained with corresponding state-of-the-art methods. Our efforts go beyond earlier surveys which are either task-specific or concentrate only on one type of visual content, i.e., image or video. Furthermore, we also provide some potential future directions in this field of research with an anticipation that this survey stimulates innovative thoughts and ideas to address the existing challenges and build new applications.

arXiv commentPublished at the Journal of Artificial Intelligence Research (JAIR); 135 pages

References in corpus (134)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Neural Machine Translation by Jointly Learning to Align and Translate
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Semi-Supervised Classification with Graph Convolutional Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Language Models are Few-Shot Learners
• The Kinetics Human Action Video Dataset
• Generative Adversarial Text to Image Synthesis
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Unsupervised Learning of Video Representations using LSTMs
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Building Natural-Language Generation Systems
• VQA: Visual Question Answering
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• A simple neural network module for relational reasoning
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Exploring Models and Data for Image Question Answering
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
• Speaker-Follower Models for Vision-and-Language Navigation
• Phrase-Based & Neural Unsupervised Machine Translation
• Coherent Multi-Sentence Video Description with Variable Level of Detail
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Learning What and Where to Draw
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• Learning to Compose and Reason with Language Tree Structures for Visual Grounding
• Learning to Count Objects in Natural Images for Visual Question Answering
• FiLM: Visual Reasoning with a General Conditioning Layer
• How2: A Large-scale Dataset for Multimodal Language Understanding
• UNITER: UNiversal Image-TExt Representation Learning
• Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space
• Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
• AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
• VideoBERT: A Joint Model for Video and Language Representation Learning
• The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
• Measuring abstract reasoning in neural networks
• Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
• Representations of language in a model of visually grounded speech signal
• Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
• Generation and Comprehension of Unambiguous Object Descriptions
• Compositional Attention Networks for Machine Reasoning
• Learning to Compose Neural Networks for Question Answering
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• TAC-GAN - Text Conditioned Auxiliary Classifier Generative Adversarial Network
• Visual Storytelling
• Imagination improves Multimodal Translation
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Aligning where to see and what to tell: image caption with region-based attention and scene factorization
• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• Multimodal Pivots for Image Caption Translation
• Contrastive Learning for Image Captioning
• Visual Reference Resolution using Attention Memory for Visual Dialog
• MAttNet: Modular Attention Network for Referring Expression Comprehension
• Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
• Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model
• DenseCap: Fully Convolutional Localization Networks for Dense Captioning
• Visual Madlibs: Fill in the blank Image Generation and Question Answering
• Unified Vision-Language Pre-Training for Image Captioning and VQA
• MirrorGAN: Learning Text-to-image Generation by Redescription
• Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language
• Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
• VinVL: Revisiting Visual Representations in Vision-Language Models
• Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems
• Stack-Captioning: Coarse-to-Fine Learning for Image Captioning
• Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network
• CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog
• Video Storytelling: Textual Summaries for Events
• TVQA: Localized, Compositional Video Question Answering
• Visual Question Answering: A Survey of Methods and Datasets
• Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis
• Object Relational Graph with Teacher-Recommended Learning for Video Captioning
• VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
• An empirical study on the effectiveness of images in Multimodal Neural Machine Translation
• Yin and Yang: Balancing and Answering Binary Visual Questions
• Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
• Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning
• Using Syntax to Ground Referring Expressions in Natural Images
• Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects
• Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering
• SentiCap: Generating Image Descriptions with Sentiments
• RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
• Generating Multiple Objects at Spatially Distinct Locations
• Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner
• Evaluation of Automatic Video Captioning Using Direct Assessment
• Language-Conditioned Graph Networks for Relational Reasoning
• Multimodal Compact Bilinear Pooling for Multimodal Neural Machine Translation
• Reasoning Visual Dialogs with Structural and Partial Observations
• TVQA+: Spatio-Temporal Grounding for Video Question Answering
• No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
• Neural Language Generation: Formulation, Methods, and Evaluation
• Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
• Video Captioning via Hierarchical Reinforcement Learning
• LanguageRefer: Spatial-Language Model for 3D Visual Grounding
• StoryGAN: A Sequential Conditional GAN for Story Visualization
• Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation
• STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset
• Temporally Grounding Language Queries in Videos by Contextual Boundary-aware Prediction
• SemStyle: Learning to Generate Stylised Image Captions using Unaligned Text
• Neural Modular Control for Embodied Question Answering
• Using Inter-Sentence Diverse Beam Search to Reduce Redundancy in Visual Storytelling
• Grounding Referring Expressions in Images by Variational Context
• Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning
• Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
• Unsupervised Multi-modal Neural Machine Translation
• COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis
• The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation
• Visual Text Correction
• Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
• OBJ2TEXT: Generating Visually Descriptive Language from Object Layouts
• Hierarchically-Attentive RNN for Album Summarization and Storytelling
• VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
• Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioning
• VSE-ens: Visual-Semantic Embeddings with Efficient Negative Sampling
• Just Ask:An Interactive Learning Framework for Vision and Language Navigation
• Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat
• Challenges and Prospects in Vision and Language Research
• Image Manipulation with Natural Language using Two-sidedAttentive Conditional Generative Adversarial Network
• Towards Better Adversarial Synthesis of Human Images from Text
• Object Counts! Bringing Explicit Detections Back into Image Captioning

Cited by in corpus (16)

• Adversarial Text-to-Image Synthesis: A Review
• Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translation
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• Levels of explainable artificial intelligence for human-aligned conversational explanations
• A Multimodal Memes Classification: A Survey and Open Research Issues
• Recent Advances and Trends in Multimodal Deep Learning: A Review
• The Next Big Thing(s) in Unsupervised Machine Learning: Five Lessons from Infant Learning
• Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approach
• Learning Cross-modal Context Graph for Visual Grounding
• Inferring spatial relations from textual descriptions of images
• Evaluating Multimodal Representations on Visual Semantic Textual Similarity
• Cross-Modal Generative Augmentation for Visual Question Answering
• PyTorchPipe: a framework for rapid prototyping of pipelines combining language and vision
• Transfer Learning in Visual and Relational Reasoning
• A Competitive Deep Neural Network Approach for the ImageCLEFmed Caption 2020 Task
• Case Relation Transformer: A Crossmodal Language Generation Model for Fetching Instructions