ImageSpirit: Verbal Guided Image Parsing

Abstract

Humans describe images in terms of nouns and adjectives while algorithms operate on images represented as sets of pixels. Bridging this gap between how humans would like to access images versus their typical representation is the goal of image parsing, which involves assigning object and attribute labels to pixel. In this paper we propose treating nouns as object labels and adjectives as visual attribute labels. This allows us to formulate the image parsing problem as one of jointly estimating per-pixel object and attribute labels from a set of training images. We propose an efficient (interactive time) solution. Using the extracted labels as handles, our system empowers a user to verbally refine the results. This enables hands-free parsing of an image into pixel-wise object/attribute labels that correspond to human semantics. Verbally selecting objects of interests enables a novel and natural interaction modality that can possibly be used to interact with new generation devices (e.g. smart phones, Google Glass, living room devices). We demonstrate our system on a large number of real-world images with varying complexity. To help understand the tradeoffs compared to traditional mouse based interactions, results are reported for both a large scale quantitative evaluation and a user study.

http://mmcheng.net/imagespirit/

References in corpus (1)

• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials

Cited by in corpus (12)

• Generation and Comprehension of Unambiguous Object Descriptions
• SemanticPaint: A Framework for the Interactive Segmentation of 3D Scenes
• Language-based Photo Color Adjustment for Graphic Designs
• Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis
• Learning to Globally Edit Images with Textual Description
• Speech-Based Visual Question Answering
• Language-Driven Image Style Transfer
• Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding
• SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning
• Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Grounding
• A Novel Technique for Evidence based Conditional Inference in Deep Neural Networks via Latent Feature Perturbation
• A Data-driven Approach for Furniture and Indoor Scene Colorization