Exposing Semantic Segmentation Failures via Maximum Discrepancy Competition

Abstract

Semantic segmentation is an extensively studied task in computer vision, with numerous methods proposed every year. Thanks to the advent of deep learning in semantic segmentation, the performance on existing benchmarks is close to saturation. A natural question then arises: Does the superior performance on the closed (and frequently re-used) test sets transfer to the open visual world with unconstrained variations? In this paper, we take steps toward answering the question by exposing failures of existing semantic segmentation methods in the open visual world under the constraint of very limited human labeling effort. Inspired by previous research on model falsification, we start from an arbitrarily large image set, and automatically sample a small image set by MAximizing the Discrepancy (MAD) between two segmentation methods. The selected images have the greatest potential in falsifying either (or both) of the two methods. We also explicitly enforce several conditions to diversify the exposed failures, corresponding to different underlying root causes. A segmentation method, whose failures are more difficult to be exposed in the MAD competition, is considered better. We conduct a thorough MAD diagnosis of ten PASCAL VOC semantic segmentation algorithms. With detailed analysis of experimental results, we point out strengths and weaknesses of the competing algorithms, as well as potential research directions for further advancement in semantic segmentation. The codes are publicly available at \url{https://github.com/QTJiebin/MAD_Segmentation}.

arXiv comment19 pages, 12 figures, 5 tables, accepted by IJCV

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Explaining and Harnessing Adversarial Examples
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Deep Domain Confusion: Maximizing for Domain Invariance
• Multi-Scale Context Aggregation by Dilated Convolutions
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Pyramid Attention Network for Semantic Segmentation
• Controversial stimuli: pitting neural networks against each other as models of human recognition
• Light-Weight RefineNet for Real-Time Semantic Segmentation
• Excessive Invariance Causes Adversarial Vulnerability
• Practical Solutions for Machine Learning Safety in Autonomous Vehicles
• I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively