IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers

Abstract

The self-attention-based model, transformer, is recently becoming the leading backbone in the field of computer vision. In spite of the impressive success made by transformers in a variety of vision tasks, it still suffers from heavy computation and intensive memory costs. To address this limitation, this paper presents an Interpretability-Aware REDundancy REDuction framework (IA-RED${}^2$). We start by observing a large amount of redundant computation, mainly spent on uncorrelated input patches, and then introduce an interpretable module to dynamically and gracefully drop these redundant patches. This novel framework is then extended to a hierarchical structure, where uncorrelated tokens at different stages are gradually removed, resulting in a considerable shrinkage of computational cost. We include extensive experiments on both image and video tasks, where our method could deliver up to 1.4x speed-up for state-of-the-art models like DeiT and TimeSformer, by only sacrificing less than 0.7% accuracy. More importantly, contrary to other acceleration approaches, our method is inherently interpretable with substantial visual evidence, making vision transformer closer to a more human-understandable architecture while being lighter. We demonstrate that the interpretability that naturally emerged in our framework can outperform the raw attention learned by the original visual transformer, as well as those generated by off-the-shelf interpretation methods, with both qualitative and quantitative results. Project Page: http://people.csail.mit.edu/bpan/ia-red/.

arXiv commentAccepted in NeurIPS 2021

References in corpus (24)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Gaussian Error Linear Units (GELUs)
• Axiomatic Attribution for Deep Networks
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Is Space-Time Attention All You Need for Video Understanding?
• Transformer in Transformer
• Linformer: Self-Attention with Linear Complexity
• SmoothGrad: removing noise by adding noise
• Adaptive Computation Time for Recurrent Neural Networks
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• CvT: Introducing Convolutions to Vision Transformers
• Conditional Computation in Neural Networks for faster models
• Pre-Trained Image Processing Transformer
• Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inference
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Full-Gradient Representation for Neural Network Visualization
• ConTNet: Why not use convolution and transformer at the same time?
• Multiscale Vision Transformers
• Dynamic Neural Networks: A Survey
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference
• Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification
• Generative Adversarial Transformers
• Patch Slimming for Efficient Vision Transformers

Cited by in corpus (5)

• Human Action Recognition from Various Data Modalities: A Review
• Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity
• Homogeneous Tokenizer Matters: Homogeneous Visual Tokenizer for Remote Sensing Image Understanding
• Automation of Quantum Dot Measurement Analysis via Explainable Machine Learning
• Efficient Video Transformers with Spatial-Temporal Token Selection