Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

Abstract

Attention plays a critical role in human visual experience. Furthermore, it has recently been demonstrated that attention can also play an important role in the context of applying artificial neural networks to a variety of tasks from fields such as computer vision and NLP. In this work we show that, by properly defining attention for convolutional neural networks, we can actually use this type of information in order to significantly improve the performance of a student CNN network by forcing it to mimic the attention maps of a powerful teacher network. To that end, we propose several novel methods of transferring attention, showing consistent improvement across a variety of datasets and convolutional neural network architectures. Code and models for our experiments are available at https://github.com/szagoruyko/attention-transfer

Cited by in corpus (215)

• Knowledge Distillation: A Survey
• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Binary Neural Networks: A Survey
• Reinforcement Learning with Augmented Data
• Adaptive Multi-Teacher Multi-level Knowledge Distillation
• Efficient Medical Image Segmentation Based on Knowledge Distillation
• Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition
• TCLR: Temporal Contrastive Learning for Video Representation
• Distilled Siamese Networks for Visual Tracking
• Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks
• Network Pruning via Transformable Architecture Search
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Learning Hierarchical Attention for Weakly-supervised Chest X-Ray Abnormality Localization and Diagnosis
• Omni-Scale Feature Learning for Person Re-Identification
• Torchreid: A Library for Deep Learning Person Re-Identification in Pytorch
• Learning Lightweight Lane Detection CNNs by Self Attention Distillation
• Fast Fine-grained Image Classification via Weakly Supervised Discriminative Localization
• Double Similarity Distillation for Semantic Image Segmentation
• Controlling an Autonomous Vehicle with Deep Reinforcement Learning
• Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation
• Hierarchical Self-supervised Augmented Knowledge Distillation
• Training Binary Neural Networks with Real-to-Binary Convolutions
• Memorizing Complementation Network for Few-Shot Class-Incremental Learning
• On Representation Knowledge Distillation for Graph Neural Networks
• Saliency for Fine-grained Object Recognition in Domains with Scarce Training Data
• MRI-based Alzheimer's disease prediction via distilling the knowledge in multi-modal data
• Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Object DGCNN: 3D Object Detection using Dynamic Graphs
• Learning Generalisable Omni-Scale Representations for Person Re-Identification
• Relational Knowledge Distillation
• Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion
• Synergic Adversarial Label Learning for Grading Retinal Diseases via Knowledge Distillation and Multi-task Learning
• Learning Student-Friendly Teacher Networks for Knowledge Distillation
• A Survey on Symbolic Knowledge Distillation of Large Language Models
• Distilling Object Detectors with Task Adaptive Regularization
• ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks
• Distilling Knowledge by Mimicking Features
• Improved training of binary networks for human pose estimation and image recognition
• Channel Distillation: Channel-Wise Attention for Knowledge Distillation
• Distilling Object Detectors with Feature Richness
• Variational Information Distillation for Knowledge Transfer
• Multi-Task Incremental Learning for Object Detection
• Zero-shot Knowledge Transfer via Adversarial Belief Matching
• Bridging the Gap Between Patient-specific and Patient-independent Seizure Prediction via Knowledge Distillation
• Towards Optimal Structured CNN Pruning via Generative Adversarial Learning
• Transfer Learning with Dynamic Adversarial Adaptation Network
• CHS-Net: A Deep learning approach for hierarchical segmentation of COVID-19 infected CT images
• Regularizing Class-wise Predictions via Self-knowledge Distillation
• ShrinkTeaNet: Million-scale Lightweight Face Recognition via Shrinking Teacher-Student Networks
• Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks
• Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regression
• Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics
• Role of Data Augmentation Strategies in Knowledge Distillation for Wearable Sensor Data
• DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer
• DBP: Discrimination Based Block-Level Pruning for Deep Model Acceleration
• Symmetric Uncertainty-Aware Feature Transmission for Depth Super-Resolution
• Radio2Text: Streaming Speech Recognition Using mmWave Radio Signals
• Doing More with Less: Overcoming Data Scarcity for POI Recommendation via Cross-Region Transfer
• Evaluating Weakly Supervised Object Localization Methods Right
• Attention-based Dropout Layer for Weakly Supervised Object Localization
• Residual Knowledge Distillation
• Label-guided Attention Distillation for Lane Segmentation
• MERANet: Facial Micro-Expression Recognition using 3D Residual Attention Network
• Multi-Label Knowledge Distillation
• Preparing Lessons: Improve Knowledge Distillation with Better Supervision
• Knowledge Amalgamation for Object Detection with Transformers
• Optimizing speed/accuracy trade-off for person re-identification via knowledge distillation
• Group channel pruning and spatial attention distilling for object detection
• Structured Knowledge Distillation for Dense Prediction
• Switchable Precision Neural Networks
• Justifying Diagnosis Decisions by Deep Neural Networks
• Knowledge distillation via adaptive instance normalization
• MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
• SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Models
• Understanding More about Human and Machine Attention in Deep Neural Networks
• Attention Convolutional Binary Neural Tree for Fine-Grained Visual Categorization
• BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget
• Maximizing Discrimination Capability of Knowledge Distillation with Energy Function
• Search for Better Students to Learn Distilled Knowledge
• An Embarrassingly Simple Approach for Knowledge Distillation
• MimicDet: Bridging the Gap Between One-Stage and Two-Stage Object Detection
• A Comprehensive Overhaul of Feature Distillation
• Multi-View Attention Transfer for Efficient Speech Enhancement
• Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation
• Does Knowledge Distillation Really Work?
• Distilling Knowledge via Knowledge Review
• Noisy Self-Knowledge Distillation for Text Summarization
• Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation
• CATFace: Cross-Attribute-Guided Transformer with Self-Attention Distillation for Low-Quality Face Recognition
• Balanced Knowledge Distillation for Long-tailed Learning
• PyNET-QxQ: An Efficient PyNET Variant for QxQ Bayer Pattern Demosaicing in CMOS Image Sensors
• Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics
• Semi-Supervised Domain Generalizable Person Re-Identification
• Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations
• Inter-Region Affinity Distillation for Road Marking Segmentation
• Learning Scene Structure Guidance via Cross-Task Knowledge Transfer for Single Depth Super-Resolution
• Efficient Semantic Video Segmentation with Per-frame Inference
• Leveraging Angular Distributions for Improved Knowledge Distillation
• Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation
• Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT
• Learning Metrics from Teachers: Compact Networks for Image Embedding
• Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation
• Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledge
• DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning
• Topological Persistence Guided Knowledge Distillation for Wearable Sensor Data
• Interactive Knowledge Distillation
• Rethinking Implicit Neural Representations for Vision Learners
• Full-Cycle Energy Consumption Benchmark for Low-Carbon Computer Vision
• Distillation Guided Residual Learning for Binary Convolutional Neural Networks
• Collaborative Distillation for Ultra-Resolution Universal Style Transfer
• NeuGuard: Lightweight Neuron-Guided Defense against Membership Inference Attacks
• PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation
• Restructuring, Pruning, and Adjustment of Deep Models for Parallel Distributed Inference
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• Attribution Preservation in Network Compression for Reliable Network Interpretation
• Localization Distillation for Dense Object Detection
• Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup
• Spatial Likelihood Voting with Self-Knowledge Distillation for Weakly Supervised Object Detection
• RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation
• Biphasic Learning of GANs for High-Resolution Image-to-Image Translation
• Knowledge Distillation Under Ideal Joint Classifier Assumption
• Distilling Ensemble of Explanations for Weakly-Supervised Pre-Training of Image Segmentation Models
• Embedding Transfer with Label Relaxation for Improved Metric Learning
• Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
• Data Efficient Stagewise Knowledge Distillation
• Learning with Privileged Information for Efficient Image Super-Resolution
• Bidirectional Knowledge Reconfiguration for Lightweight Point Cloud Analysis
• Cross-Layer Distillation with Semantic Calibration
• S2SD: Simultaneous Similarity-based Self-Distillation for Deep Metric Learning
• Distilling with Performance Enhanced Students
• Periocular Embedding Learning with Consistent Knowledge Distillation from Face
• Knowledge Transfer via Dense Cross-Layer Mutual-Distillation
• Distilling Audio-Visual Knowledge by Compositional Contrastive Learning
• Extreme Low Resolution Activity Recognition with Confident Spatial-Temporal Attention Transfer
• Knowledge Distillation in Document Retrieval
• Separable Layers Enable Structured Efficient Linear Substitutions
• LEA-Net: Layer-wise External Attention Network for Efficient Color Anomaly Detection
• Parameter Efficient Deep Neural Networks with Bilinear Projections
• LNPT: Label-free Network Pruning and Training
• Training convolutional neural networks with cheap convolutions and online distillation
• Robustness and Diversity Seeking Data-Free Knowledge Distillation
• Distribution-sensitive Information Retention for Accurate Binary Neural Network
• DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck
• Graph-Free Knowledge Distillation for Graph Neural Networks
• Student Network Learning via Evolutionary Knowledge Distillation
• Fixing the Teacher-Student Knowledge Discrepancy in Distillation
• BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements
• Text is Text, No Matter What: Unifying Text Recognition using Knowledge Distillation
• Learning from a Lightweight Teacher for Efficient Knowledge Distillation
• Information Theoretic Representation Distillation
• Oracle Teacher: Leveraging Target Information for Better Knowledge Distillation of CTC Models
• Knowledge Distillation Meets Self-Supervision
• Black-Box Dissector: Towards Erasing-based Hard-Label Model Stealing Attack
• Tatum-Level Drum Transcription Based on a Convolutional Recurrent Neural Network with Language Model-Based Regularized Training
• Matching Guided Distillation
• Dual-path CNN with Max Gated block for Text-Based Person Re-identification
• TinyGAN: Distilling BigGAN for Conditional Image Generation
• Distill-2MD-MTL: Data Distillation based on Multi-Dataset Multi-Domain Multi-Task Frame Work to Solve Face Related Tasksks, Multi Task Learning, Semi-Supervised Learning
• Balanced Binary Neural Networks with Gated Residual
• Training Quantized Neural Networks with a Full-precision Auxiliary Module
• Learning Interpretable End-to-End Vision-Based Motion Planning for Autonomous Driving with Optical Flow Distillation
• Metric Learning for Anti-Compression Facial Forgery Detection
• Neural Parameter Allocation Search
• Introspective Learning by Distilling Knowledge from Online Self-explanation
• Orderly Dual-Teacher Knowledge Distillation for Lightweight Human Pose Estimation
• Prime-Aware Adaptive Distillation
• DEPARA: Deep Attribution Graph for Deep Knowledge Transferability
• RDPD: Rich Data Helps Poor Data via Imitation
• Dynamic Slimmable Denoising Network
• Rapid Elastic Architecture Search under Specialized Classes and Resource Constraints
• Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation
• Arch-Net: Model Distillation for Architecture Agnostic Model Deployment
• Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation
• Semi-Online Knowledge Distillation
• Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learning
• Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation
• Knowledge Transfer Graph for Deep Collaborative Learning
• Training a Binary Weight Object Detector by Knowledge Transfer for Autonomous Driving
• EEG aided boosting of single-lead ECG based sleep staging with Deep Knowledge Distillation
• Pool of Experts: Realtime Querying Specialized Knowledge in Massive Neural Networks
• Creating Lightweight Object Detectors with Model Compression for Deployment on Edge Devices
• Zero-shot Adversarial Quantization
• Measure Twice, Cut Once: Quantifying Bias and Fairness in Deep Neural Networks
• Deep Ensemble Collaborative Learning by using Knowledge-transfer Graph for Fine-grained Object Classification
• LabelEnc: A New Intermediate Supervision Method for Object Detection
• Diversified Mutual Learning for Deep Metric Learning
• AIP: Adversarial Iterative Pruning Based on Knowledge Transfer for Convolutional Neural Networks
• Transferring Inter-Class Correlation
• Multi-Scale Aligned Distillation for Low-Resolution Detection
• SERE: Exploring Feature Self-relation for Self-supervised Transformer
• CHEER: Rich Model Helps Poor Model via Knowledge Infusion
• Multilingual AMR Parsing with Noisy Knowledge Distillation
• Ensemble Knowledge Distillation for CTR Prediction
• Locally Linear Region Knowledge Distillation
• A Computer Vision Approach to Combat Lyme Disease
• Distilling Visual Priors from Self-Supervised Learning
• Unsupervised Domain Adaptive Person Re-Identification via Human Learning Imitation
• Pacemaker: Intermediate Teacher Knowledge Distillation For On-The-Fly Convolutional Neural Network
• Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation
• Topology Distillation for Recommender System
• KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation
• Distilling Image Classifiers in Object Detectors
• Be Your Own Best Competitor! Multi-Branched Adversarial Knowledge Transfer
• Confidence Conditioned Knowledge Distillation
• On the Efficiency of Subclass Knowledge Distillation in Classification Tasks
• GenURL: A General Framework for Unsupervised Representation Learning
• Collaborative Group Learning
• 3D attention mechanism for fine-grained classification of table tennis strokes using a Twin Spatio-Temporal Convolutional Neural Networks
• A Survey on Green Deep Learning
• Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition
• A One-step Pruning-recovery Framework for Acceleration of Convolutional Neural Networks