Inverse Problems Leveraging Pre-trained Contrastive Representations

Abstract

We study a new family of inverse problems for recovering representations of corrupted data. We assume access to a pre-trained representation learning network R(x) that operates on clean images, like CLIP. The problem is to recover the representation of an image R(x), if we are only given a corrupted version A(x), for some known forward operator A. We propose a supervised inversion method that uses a contrastive objective to obtain excellent representations for highly corrupted images. Using a linear probe on our robust representations, we achieve a higher accuracy than end-to-end supervised baselines when classifying images with various types of distortions, including blurring, additive noise, and random pixel masking. We evaluate on a subset of ImageNet and observe that our method is robust to varying levels of distortion. Our method outperforms end-to-end baselines even with a fraction of the labeled data in a wide range of forward operators.

arXiv commentInitial version. Final version to appear in Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS 2021)

References in corpus (11)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• Learning Transferable Visual Models From Natural Language Supervision
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• COVID-19 Image Data Collection
• Contrastive Multiview Coding
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Intriguing Properties of Contrastive Losses
• Understanding Self-supervised Learning with Dual Deep Networks
• Task-Aware Compressed Sensing with Generative Adversarial Networks