Vision and Language: from Visual Perception to Content Creation

Abstract

Vision and language are two fundamental capabilities of human intelligence. Humans routinely perform tasks through the interactions between vision and language, supporting the uniquely human capacity to talk about what they see or hallucinate a picture on a natural-language description. The valid question of how language interacts with vision motivates us researchers to expand the horizons of computer vision area. In particular, "vision to language" is probably one of the most popular topics in the past five years, with a significant growth in both volume of publications and extensive applications, e.g., captioning, visual question answering, visual dialog, language navigation, etc. Such tasks boost visual perception with more comprehensive understanding and diverse linguistic representations. Going beyond the progresses made in "vision to language," language can also contribute to vision understanding and offer new possibilities of visual content creation, i.e., "language to vision." The process performs as a prism through which to create visual content conditioning on the language inputs. This paper reviews the recent advances along these two dimensions: "vision to language" and "language to vision." More concretely, the former mainly focuses on the development of image/video captioning, as well as typical encoder-decoder structures and benchmarks, while the latter summarizes the technologies of visual content creation. The real-world deployment or services of vision and language are elaborated as well.

References in corpus (14)

• Conditional Generative Adversarial Nets
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Generative Adversarial Text to Image Synthesis
• Pose Guided Person Image Generation
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
• High-Fidelity Image Generation With Fewer Labels
• Review Networks for Caption Generation
• Rich Image Captioning in the Wild
• Soft-Gated Warping-GAN for Pose-Guided Person Image Synthesis
• Generating Multiple Objects at Spatially Distinct Locations
• Photo-Realistic Facial Details Synthesis from Single Image
• Relationship-Aware Spatial Perception Fusion for Realistic Scene Layout Generation