Sparse R-CNN: End-to-End Object Detection with Learnable Proposals

Abstract

We present Sparse R-CNN, a purely sparse method for object detection in images. Existing works on object detection heavily rely on dense object candidates, such as $k$ anchor boxes pre-defined on all grids of image feature map of size $H\times W$. In our method, however, a fixed sparse set of learned object proposals, total length of $N$, are provided to object recognition head to perform classification and location. By eliminating $HWk$ (up to hundreds of thousands) hand-designed object candidates to $N$ (e.g. 100) learnable proposals, Sparse R-CNN completely avoids all efforts related to object candidates design and many-to-one label assignment. More importantly, final predictions are directly output without non-maximum suppression post-procedure. Sparse R-CNN demonstrates accuracy, run-time and training convergence performance on par with the well-established detector baselines on the challenging COCO dataset, e.g., achieving 45.0 AP in standard $3\times$ training schedule and running at 22 fps using ResNet-50 FPN model. We hope our work could inspire re-thinking the convention of dense prior in object detectors. The code is available at: https://github.com/PeizeSun/SparseR-CNN.

add test-dev; add crowdhuman

References in corpus (16)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Decoupled Weight Decay Regularization
• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• CrowdHuman: A Benchmark for Detecting Human in a Crowd
• DenseBox: Unifying Landmark Localization with End to End Object Detection
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• Scaled-YOLOv4: Scaling Cross Stage Partial Network
• DETR for Crowd Pedestrian Detection
• Conditional Convolutions for Instance Segmentation
• Spatially Consistent Representation Learning

Cited by in corpus (15)

• PVT v2: Improved Baselines with Pyramid Vision Transformer
• TransTrack: Multiple Object Tracking with Transformer
• Focal Self-attention for Local-Global Interactions in Vision Transformers
• Efficient DETR: Improving End-to-End Object Detector with Dense Prior
• Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer
• FoodLogoDet-1500: A Dataset for Large-Scale Food Logo Detection via Multi-Scale Feature Decoupling Network
• Objects in Semantic Topology
• End-to-End Video Object Detection with Spatial-Temporal Transformers
• Towards High-Quality Temporal Action Detection with Sparse Proposals
• Structured Sparse R-CNN for Direct Scene Graph Generation
• Augmenting Proposals by the Detector Itself
• The MIS Check-Dam Dataset for Object Detection and Instance Segmentation Tasks
• Gaze Estimation using Transformer
• Discriminative Semantic Feature Pyramid Network with Guided Anchoring for Logo Detection
• Suppress-and-Refine Framework for End-to-End 3D Object Detection