Adversarial Structure Matching for Structured Prediction Tasks

Abstract

Pixel-wise losses, e.g., cross-entropy or L2, have been widely used in structured prediction tasks as a spatial extension of generic image classification or regression. However, its i.i.d. assumption neglects the structural regularity present in natural images. Various attempts have been made to incorporate structural reasoning mostly through structure priors in a cooperative way where co-occurring patterns are encouraged. We, on the other hand, approach this problem from an opposing angle and propose a new framework, Adversarial Structure Matching (ASM), for training such structured prediction networks via an adversarial process, in which we train a structure analyzer that provides the supervisory signals, the ASM loss. The structure analyzer is trained to maximize the ASM loss, or to emphasize recurring multi-scale hard negative structural mistakes among co-occurring patterns. On the contrary, the structured prediction network is trained to reduce those mistakes and is thus enabled to distinguish fine-grained structures. As a result, training structured prediction networks using ASM reduces contextual confusion among objects and improves boundary localization. We demonstrate that our ASM outperforms pixel-wise IID loss or structural prior GAN loss on three different structured prediction tasks: semantic segmentation, monocular depth estimation, and surface normal prediction.

arXiv commentIn CVPR 2019. Webpage & Code: https://jyhjinghwang.github.io/projects/asm.html

References in corpus (12)

• Rethinking Atrous Convolution for Semantic Image Segmentation
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Conditional Random Fields as Recurrent Neural Networks
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling
• Semantic Segmentation using Adversarial Networks
• Indoor Semantic Segmentation using depth information
• Learning Deep Structured Models
• Pixel-wise Deep Learning for Contour Detection
• Deep Value Networks Learn to Evaluate and Iteratively Refine Structured Outputs