YOLACT++: Better Real-time Instance Segmentation

Abstract

We present a simple, fully-convolutional model for real-time (>30 fps) instance segmentation that achieves competitive results on MS COCO evaluated on a single Titan Xp, which is significantly faster than any previous state-of-the-art approach. Moreover, we obtain this result after training on only one GPU. We accomplish this by breaking instance segmentation into two parallel subtasks: (1) generating a set of prototype masks and (2) predicting per-instance mask coefficients. Then we produce instance masks by linearly combining the prototypes with the mask coefficients. We find that because this process doesn't depend on repooling, this approach produces very high-quality masks and exhibits temporal stability for free. Furthermore, we analyze the emergent behavior of our prototypes and show they learn to localize instances on their own in a translation variant manner, despite being fully-convolutional. We also propose Fast NMS, a drop-in 12 ms faster replacement for standard NMS that only has a marginal performance penalty. Finally, by incorporating deformable convolutions into the backbone network, optimizing the prediction head with better anchor scales and aspect ratios, and adding a novel fast mask re-scoring branch, our YOLACT++ model can achieve 34.1 mAP on MS COCO at 33.5 fps, which is fairly close to the state-of-the-art approaches while still running at real-time.

arXiv commentJournal extension of our previous conference paper arXiv:1904.02689

References in corpus (9)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• YOLOv3: An Incremental Improvement
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• Focal Loss for Dense Object Detection
• ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
• Associative Embedding: End-to-End Learning for Joint Detection and Grouping
• Semantic Instance Segmentation with a Discriminative Loss Function
• Semantic Instance Segmentation via Deep Metric Learning
• RetinaMask: Learning to predict masks improves state-of-the-art single-shot detection for free

Cited by in corpus (34)

• Masked-attention Mask Transformer for Universal Image Segmentation
• PhenoBench -- A Large Dataset and Benchmarks for Semantic Image Interpretation in the Agricultural Domain
• Tomato Maturity Recognition with Convolutional Transformers
• Panoptic Segmentation Meets Remote Sensing
• BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation
• A Comprehensive Review of Modern Object Segmentation Approaches
• Development and evaluation of automated localisation and reconstruction of all fruits on tomato plants in a greenhouse based on multi-view perception and 3D multi-object tracking
• SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation
• mBEST: Realtime Deformable Linear Object Detection Through Minimal Bending Energy Skeleton Pixel Traversals
• Assessing YOLACT++ for real time and robust instance segmentation of medical instruments in endoscopic procedures
• Online Object-Oriented Semantic Mapping and Map Updating
• On the use of neural networks for the structural characterization of polymeric porous materials
• FollowMe: a Robust Person Following Framework Based on Re-Identification and Gestures
• ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection
• Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation
• Benchmarking the Robustness of Instance Segmentation Models
• Vision and Tactile Robotic System to Grasp Litter in Outdoor Environments
• End-to-End Human Instance Matting
• DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision
• Deep Instance Segmentation and Visual Servoing to Play Jenga with a Cost-Effective Robotic System
• Soft Constrained Autonomous Vehicle Navigation using Gaussian Processes and Instance Segmentation
• Pointly-Supervised Instance Segmentation
• Deep Variational Instance Segmentation
• Analysis of Deep-Learning Methods in an ISO/TS 15066-Compliant Human-Robot Safety Framework
• Crossover Learning for Fast Online Video Instance Segmentation
• Trapped in texture bias? A large scale comparison of deep instance segmentation
• YolactEdge: Real-time Instance Segmentation on the Edge
• PlaneSegNet: Fast and Robust Plane Estimation Using a Single-stage Instance Segmentation CNN
• INSTA-YOLO: Real-Time Instance Segmentation
• BEyond observation: an approach for ObjectNav
• Semantic Dense Reconstruction with Consistent Scene Segments
• Panoptic Segmentation: A Review
• DIY Graphics Tab: A Cost-Effective Alternative to Graphics Tablet for Educators
• Semi-Supervised Surface Anomaly Detection of Composite Wind Turbine Blades From Drone Imagery