Learning Image Representations by Completing Damaged Jigsaw Puzzles

Abstract

In this paper, we explore methods of complicating self-supervised tasks for representation learning. That is, we do severe damage to data and encourage a network to recover them. First, we complicate each of three powerful self-supervised task candidates: jigsaw puzzle, inpainting, and colorization. In addition, we introduce a novel complicated self-supervised task called "Completing damaged jigsaw puzzles" which is puzzles with one piece missing and the other pieces without color. We train a convolutional neural network not only to solve the puzzles, but also generate the missing content and colorize the puzzles. The recovery of the aforementioned damage pushes the network to obtain robust and general-purpose representations. We demonstrate that complicating the self-supervised tasks improves their original versions and that our final task learns more robust and transferable representations compared to the previous methods, as well as the simple combination of our candidate tasks. Our approach achieves state-of-the-art performance in transfer learning on PASCAL classification and semantic segmentation.

arXiv commentaccepted at WACV 2018

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Unsupervised Visual Representation Learning by Context Prediction
• Unsupervised Learning of Visual Representations using Videos
• Data-dependent Initializations of Convolutional Neural Networks
• Learning to See by Moving
• Learning visual groups from co-occurrences in space and time
• Colorization as a Proxy Task for Visual Understanding
• ContextLocNet: Context-Aware Deep Network Models for Weakly Supervised Localization
• Two-Phase Learning for Weakly Supervised Object Localization
• Deep Self-Taught Learning for Weakly Supervised Object Localization
• Transitive Invariance for Self-supervised Visual Representation Learning

Cited by in corpus (8)

• Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
• Contrastive Learning with Adversarial Examples
• Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles
• Jigsaw Clustering for Unsupervised Visual Representation Learning
• TVDIM: Enhancing Image Self-Supervised Pretraining via Noisy Text Data
• Synthesizing Photorealistic Images with Deep Generative Learning
• Pretext Tasks selection for multitask self-supervised speech representation learning
• Exploit Clues from Views: Self-Supervised and Regularized Learning for Multiview Object Recognition