Deep Bilateral Learning for Real-Time Image Enhancement

Abstract

Performance is a critical challenge in mobile image processing. Given a reference imaging pipeline, or even human-adjusted pairs of images, we seek to reproduce the enhancements and enable real-time evaluation. For this, we introduce a new neural network architecture inspired by bilateral grid processing and local affine color transforms. Using pairs of input/output images, we train a convolutional neural network to predict the coefficients of a locally-affine model in bilateral space. Our architecture learns to make local, global, and content-dependent decisions to approximate the desired image transformation. At runtime, the neural network consumes a low-resolution version of the input image, produces a set of affine transformations in bilateral space, upsamples those transformations in an edge-preserving fashion using a new slicing node, and then applies those upsampled transformations to the full-resolution image. Our algorithm processes high-resolution images on a smartphone in milliseconds, provides a real-time viewfinder at 1080p resolution, and matches the quality of state-of-the-art approximation techniques on a large class of image operators. Unlike previous work, our model is trained off-line from data and therefore does not require access to the original operator at runtime. This allows our model to learn complex, scene-dependent transformations for which no reference implementation is available, such as the photographic edits of a human retoucher.

12 pages, 14 figures, Siggraph 2017

References in corpus (4)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

Cited by in corpus (16)

• Deep Learning Microscopy
• Single Image Portrait Relighting
• VisionISP: Repurposing the Image Signal Processor for Computer Vision Applications
• DeepFlash: Turning a Flash Selfie into a Studio Portrait
• Learning Digital Camera Pipeline for Extreme Low-Light Imaging
• Unpaired Image Enhancement Featuring Reinforcement-Learning-Controlled Image Editing Software
• Dual Illumination Estimation for Robust Exposure Correction
• Divide-and-Conquer Adversarial Learning for High-Resolution Image and Video Enhancement
• A General Decoupled Learning Framework for Parameterized Image Operators
• PixelRL: Fully Convolutional Network with Reinforcement Learning for Image Processing
• Synthetic Defocus and Look-Ahead Autofocus for Casual Videography
• Underexposed Image Correction via Hybrid Priors Navigated Deep Propagation
• Supervised and Unsupervised Learning of Parameterized Color Enhancement
• A Machine Learning Imaging Core using Separable FIR-IIR Filters
• Neural Imaging Pipelines - the Scourge or Hope of Forensics?
• Smart, Deep Copy-Paste