Pre-Trained Image Processing Transformer

Abstract

As the computing power of modern hardware is increasing strongly, pre-trained deep learning models (e.g., BERT, GPT-3) learned on large-scale datasets have shown their effectiveness over conventional methods. The big progress is mainly contributed to the representation ability of transformer and its variant architectures. In this paper, we study the low-level computer vision task (e.g., denoising, super-resolution and deraining) and develop a new pre-trained model, namely, image processing transformer (IPT). To maximally excavate the capability of transformer, we present to utilize the well-known ImageNet benchmark for generating a large amount of corrupted image pairs. The IPT model is trained on these images with multi-heads and multi-tails. In addition, the contrastive learning is introduced for well adapting to different image processing tasks. The pre-trained model can therefore efficiently employed on desired task after fine-tuning. With only one pre-trained model, IPT outperforms the current state-of-the-art methods on various low-level benchmarks. Code is available at https://github.com/huawei-noah/Pretrained-IPT and https://gitee.com/mindspore/mindspore/tree/master/model_zoo/research/cv/IPT

arXiv commentCVPR 2021

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• A Simple Framework for Contrastive Learning of Visual Representations
• DehazeNet: An End-to-End System for Single Image Haze Removal
• Language Models are Few-Shot Learners
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration
• Clearing the Skies: A deep network architecture for single-image rain removal
• OCNet: Object Context Network for Scene Parsing
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• Residual Dense Network for Image Super-Resolution
• An All-in-One Network for Dehazing and Beyond
• Cross-Scale Internal Graph Neural Network for Image Super-Resolution
• Learning Deep Transformer Models for Machine Translation
• LatentGNN: Learning Efficient Non-local Relations for Visual Recognition
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Blur More To Deblur Better: Multi-Blur2Deblur For Efficient Video Deblurring
• End-to-end Interpretable Learning of Non-blind Image Deblurring

Cited by in corpus (50)

• Transformers in Vision: A Survey
• Twins: Revisiting the Design of Spatial Attention in Vision Transformers
• Conditional Positional Encodings for Vision Transformers
• DeepViT: Towards Deeper Vision Transformer
• LocalViT: Analyzing Locality in Vision Transformers
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• CvT: Introducing Convolutions to Vision Transformers
• ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias
• ResT: An Efficient Transformer for Visual Recognition
• All Tokens Matter: Token Labeling for Training Better Vision Transformers
• Query2Label: A Simple Transformer Way to Multi-Label Classification
• CPTR: Full Transformer Network for Image Captioning
• CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
• Efficient Self-supervised Vision Transformers for Representation Learning
• ViTGAN: Training GANs with Vision Transformers
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• CMT: Convolutional Neural Networks Meet Vision Transformers
• TFPose: Direct Human Pose Estimation with Transformers
• CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation
• SOLQ: Segmenting Objects by Learning Queries
• Refiner: Refining Self-attention for Vision Transformers
• Segmenting Transparent Object in the Wild with Transformer
• Local-to-Global Self-Attention in Vision Transformers
• Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training
• Fine-grained Semantic Alignment Network for Weakly Supervised Temporal Language Grounding
• TokenPose: Learning Keypoint Tokens for Human Pose Estimation
• MOTR: End-to-End Multiple-Object Tracking with Transformer
• CoSformer: Detecting Co-Salient Object with Transformers
• Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
• Eformer: Edge Enhancement based Transformer for Medical Image Denoising
• Patch Slimming for Efficient Vision Transformers
• Delving Deep into the Generalization of Vision Transformers under Distribution Shifts
• TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding
• Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning
• CoNeS: Conditional neural fields with shift modulation for multi-sequence MRI translation
• Split and Connect: A Universal Tracklet Booster for Multi-Object Tracking
• Perceptual Image Quality Assessment with Transformers
• Post-Training Quantization for Vision Transformer
• GLiT: Neural Architecture Search for Global and Local Image Transformer
• Armour: Generalizable Compact Self-Attention for Vision Transformers
• 3D Human Texture Estimation from a Single Image with Transformers
• MUSIQ: Multi-scale Image Quality Transformer
• TED-net: Convolution-free T2T Vision Transformer-based Encoder-decoder Dilation network for Low-dose CT Denoising
• Augmented Shortcuts for Vision Transformers
• Guiding Query Position and Performing Similar Attention for Transformer-Based Detection Heads
• Ripple Attention for Visual Perception with Sub-quadratic Complexity
• Vision Pair Learning: An Efficient Training Framework for Image Classification
• A Survey on Green Deep Learning