End-to-End Object Detection with Adaptive Clustering Transformer

Abstract

End-to-end Object Detection with Transformer (DETR)proposes to perform object detection with Transformer and achieve comparable performance with two-stage object detection like Faster-RCNN. However, DETR needs huge computational resources for training and inference due to the high-resolution spatial input. In this paper, a novel variant of transformer named Adaptive Clustering Transformer(ACT) has been proposed to reduce the computation cost for high-resolution input. ACT cluster the query features adaptively using Locality Sensitive Hashing (LSH) and ap-proximate the query-key interaction using the prototype-key interaction. ACT can reduce the quadratic O(N2) complexity inside self-attention into O(NK) where K is the number of prototypes in each layer. ACT can be a drop-in module replacing the original self-attention module without any training. ACT achieves a good balance between accuracy and computation cost (FLOPs). The code is available as supplementary for the ease of experiment replication and verification. Code is released at \url{https://github.com/gaopengcuhk/SMCA-DETR/}

arXiv commentBMVC 2021 Oral

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Distilling the Knowledge in a Neural Network
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Rethinking Attention with Performers
• Asymmetric Non-local Neural Networks for Semantic Segmentation
• Multi-modality Latent Interaction Network for Visual Question Answering

Cited by in corpus (21)

• A Survey on Visual Transformer
• Focal Self-attention for Local-Global Interactions in Vision Transformers
• Light Field Image Super-Resolution with Transformers
• CvT: Introducing Convolutions to Vision Transformers
• All Tokens Matter: Token Labeling for Training Better Vision Transformers
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Efficient Self-supervised Vision Transformers for Representation Learning
• A Comprehensive Review of Modern Object Segmentation Approaches
• A Survey of Visual Transformers
• Container: Context Aggregation Network
• Refiner: Refining Self-attention for Vision Transformers
• Dynamic Label Assignment for Object Detection by Combining Predicted IoUs and Anchor IoUs
• Meta-DETR: Image-Level Few-Shot Object Detection with Inter-Class Correlation Exploitation
• Fast Convergence of DETR with Spatially Modulated Co-Attention
• FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting
• RomeBERT: Robust Training of Multi-Exit BERT
• PnP-DETR: Towards Efficient Visual Analysis with Transformers
• Sampling Equivariant Self-attention Networks for Object Detection in Aerial Images
• Probabilistic Ranking-Aware Ensembles for Enhanced Object Detections
• OadTR: Online Action Detection with Transformers
• Guiding Query Position and Performing Similar Attention for Transformer-Based Detection Heads