RelationNet++: Bridging Visual Representations for Object Detection via Transformer Decoder

Abstract

Existing object detection frameworks are usually built on a single format of object/part representation, i.e., anchor/proposal rectangle boxes in RetinaNet and Faster R-CNN, center points in FCOS and RepPoints, and corner points in CornerNet. While these different representations usually drive the frameworks to perform well in different aspects, e.g., better classification or finer localization, it is in general difficult to combine these representations in a single framework to make good use of each strength, due to the heterogeneous or non-grid feature extraction by different representations. This paper presents an attention-based decoder module similar as that in Transformer~\cite{vaswani2017attention} to bridge other representations into a typical object detector built on a single representation format, in an end-to-end fashion. The other representations act as a set of \emph{key} instances to strengthen the main \emph{query} representation features in the vanilla detectors. Novel techniques are proposed towards efficient computation of the decoder module, including a \emph{key sampling} approach and a \emph{shared location embedding} approach. The proposed module is named \emph{bridging visual representations} (BVR). It can perform in-place and we demonstrate its broad effectiveness in bridging other representations into prevalent object detection frameworks, including RetinaNet, Faster R-CNN, FCOS and ATSS, where about $1.5\sim 3.0$ AP improvements are achieved. In particular, we improve a state-of-the-art framework with a strong backbone by about $2.0$ AP, reaching $52.7$ AP on COCO test-dev. The resulting network is named RelationNet++. The code will be available at https://github.com/microsoft/RelationNet2.

NeurIPS2020 Spotlight

References in corpus (16)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• YOLOv3: An Incremental Improvement
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• FCOS: Fully Convolutional One-Stage Object Detection
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Deformable Convolutional Networks
• CornerNet-Lite: Efficient Keypoint Based Object Detection
• RepPoints: Point Set Representation for Object Detection
• Bottom-up Object Detection by Grouping Extreme and Center Points
• RepPoints V2: Verification Meets Regression for Object Detection
• Region Proposal by Guided Anchoring
• CentripetalNet: Pursuing High-quality Keypoint Pairs for Object Detection
• Dense RepPoints: Representing Visual Objects with Dense Point Sets
• Point-Set Anchors for Object Detection, Instance Segmentation and Pose Estimation
• DeNet: Scalable Real-time Object Detection with Directed Sparse Sampling
• Learning Region Features for Object Detection

Cited by in corpus (2)

• CMT: Convolutional Neural Networks Meet Vision Transformers
• Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning