Crowdsourcing in Computer Vision

Abstract

Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts. Crowdsourcing platforms offer an inexpensive method to capture human knowledge and understanding, for a vast number of visual perception tasks. In this survey, we describe the types of annotations computer vision researchers have collected using crowdsourcing, and how they have ensured that this data is of high quality while annotation effort is minimized. We begin by discussing data collection on both classic (e.g., object recognition) and recent (e.g., visual story-telling) vision tasks. We then summarize key design decisions for creating effective data collection interfaces and workflows, and present strategies for intelligently selecting the most important data instances to annotate. Finally, we conclude with some thoughts on the future of crowdsourcing in computer vision.

A 69-page meta review of the field, Foundations and Trends in Computer Graphics and Vision, 2016

References in corpus (6)

• Adaptively Learning the Crowd Kernel
• Understanding Image Virality
• Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos
• WhittleSearch: Interactive Image Search with Relative Attribute Feedback
• Discovering Attribute Shades of Meaning with the Crowd
• EventNet: A Large Scale Structured Concept Library for Complex Event Detection in Video

Cited by in corpus (3)

• Geometry in Active Learning for Binary and Multi-class Image Segmentation
• Monte Carlo Physarum Machine: Characteristics of Pattern Formation in Continuous Stochastic Transport Networks
• Towards large-scale, automated, accurate detection of CCTV camera objects using computer vision. Applications and implications for privacy, safety, and cybersecurity. (Preprint)