Colorful Image Colorization

Abstract

Given a grayscale photograph as input, this paper attacks the problem of hallucinating a plausible color version of the photograph. This problem is clearly underconstrained, so previous approaches have either relied on significant user interaction or resulted in desaturated colorizations. We propose a fully automatic approach that produces vibrant and realistic colorizations. We embrace the underlying uncertainty of the problem by posing it as a classification task and use class-rebalancing at training time to increase the diversity of colors in the result. The system is implemented as a feed-forward pass in a CNN at test time and is trained on over a million color images. We evaluate our algorithm using a "colorization Turing test," asking human participants to choose between a generated and ground truth color image. Our method successfully fools humans on 32% of the trials, significantly higher than previous methods. Moreover, we show that colorization can be a powerful pretext task for self-supervised feature learning, acting as a cross-channel encoder. This approach results in state-of-the-art performance on several feature learning benchmarks.

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Adversarial Feature Learning
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Data-dependent Initializations of Convolutional Neural Networks
• Learning Representations for Automatic Colorization
• Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles

Cited by in corpus (9)

• Density estimation using Real NVP
• Learning to Perform Physics Experiments via Deep Reinforcement Learning
• End-to-end Learning of Driving Models from Large-scale Video Datasets
• Weakly- and Self-Supervised Learning for Content-Aware Deep Image Retargeting
• Painting baryons onto N-body simulations of galaxy clusters with image-to-image deep learning
• Better Conditional Density Estimation for Neural Networks
• Generalized Deep Image to Image Regression
• SAR Image Colorization: Converting Single-Polarization to Fully Polarimetric Using Deep Neural Networks
• Pano2Vid: Automatic Cinematography for Watching 360${}^{\circ }$ Videos