Bridging the Gap Between Computational Photography and Visual Recognition

Abstract

What is the current state-of-the-art for image restoration and enhancement applied to degraded images acquired under less than ideal circumstances? Can the application of such algorithms as a pre-processing step to improve image interpretability for manual analysis or automatic visual recognition to classify scene content? While there have been important advances in the area of computational photography to restore or enhance the visual quality of an image, the capabilities of such techniques have not always translated in a useful way to visual recognition tasks. Consequently, there is a pressing need for the development of algorithms that are designed for the joint problem of improving visual appearance and recognition, which will be an enabling factor for the deployment of visual recognition tools in many real-world scenarios. To address this, we introduce the UG^2 dataset as a large-scale benchmark composed of video imagery captured under challenging conditions, and two enhancement tasks designed to test algorithmic impact on visual quality and automatic object recognition. Furthermore, we propose a set of metrics to evaluate the joint improvement of such tasks as well as individual algorithmic advances, including a novel psychophysics-based evaluation regime for human assessment and a realistic set of quantitative measures for object recognition performance. We introduce six new algorithms for image restoration or enhancement, which were created as part of the IARPA sponsored UG^2 Challenge workshop held at CVPR 2018. Under the proposed evaluation regime, we present an in-depth analysis of these algorithms and a host of deep learning-based and classic baseline approaches. From the observed results, it is evident that we are in the early days of building a bridge between computational photography and visual recognition, leaving many opportunities for innovation in this area.

arXiv commentCVPR Prize Challenge: http://www.ug2challenge.org

References in corpus (21)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections
• Toward Multimodal Image-to-Image Translation
• The Perception-Distortion Tradeoff
• Deep Bilateral Learning for Real-Time Image Enhancement
• Deep Image Prior
• Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections
• Vision Meets Drones: A Challenge
• Lucky Imaging: High Angular Resolution Imaging in the Visible from the Ground
• Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution
• iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images
• PsyPhy: A Psychophysics Driven Evaluation Framework for Visual Recognition
• Task-Driven Super Resolution: Object Detection in Low-resolution Images
• An empirical study on the effects of different types of noise in image classification tasks
• Deep Video Deblurring
• Deep Joint Face Hallucination and Recognition
• UG^2: a Video Benchmark for Assessing the Impact of Image Restoration and Enhancement on Automatic Visual Recognition
• Report on UG^2+ Challenge Track 1: Assessing Algorithms to Improve Video Object Detection and Classification from Unconstrained Mobility Platforms

Cited by in corpus (10)

• All-In-One Underwater Image Enhancement using Domain-Adversarial Learning
• UG${}^{2+}$ Track 2: A Collective Benchmark Effort for Evaluating and Advancing Image Understanding in Poor Visibility Environments
• On The Classification-Distortion-Perception Tradeoff
• Segmentation-Aware Image Denoising without Knowing True Segmentation
• Representing Noisy Image Without Denoising
• Validation of object detection in UAV-based images using synthetic data
• Delving into Robust Object Detection from Unmanned Aerial Vehicles: A Deep Nuisance Disentanglement Approach
• Report on UG^2+ Challenge Track 1: Assessing Algorithms to Improve Video Object Detection and Classification from Unconstrained Mobility Platforms
• DAVID: Dual-Attentional Video Deblurring
• LEGAN: Disentangled Manipulation of Directional Lighting and Facial Expressions by Leveraging Human Perceptual Judgements