Deep Consensus Learning

Abstract

Both generative learning and discriminative learning have recently witnessed remarkable progress using Deep Neural Networks (DNNs). For structured input synthesis and structured output prediction problems (e.g., layout-to-image synthesis and image semantic segmentation respectively), they often are studied separately. This paper proposes deep consensus learning (DCL) for joint layout-to-image synthesis and weakly-supervised image semantic segmentation. The former is realized by a recently proposed LostGAN approach, and the latter by introducing an inference network as the third player joining the two-player game of LostGAN. Two deep consensus mappings are exploited to facilitate training the three networks end-to-end: Given an input layout (a list of object bounding boxes), the generator generates a mask (label map) and then use it to help synthesize an image. The inference network infers the mask for the synthesized image. Then, the latent consensus is measured between the mask generated by the generator and the one inferred by the inference network. For the real image corresponding to the input layout, its mask also is computed by the inference network, and then used by the generator to reconstruct the real image. Then, the data consensus is measured between the real image and its reconstructed image. The discriminator still plays the role of an adversary by computing the realness scores for a real image, its reconstructed image and a synthesized image. In experiments, our DCL is tested in the COCO-Stuff dataset. It obtains compelling layout-to-image synthesis results and weakly-supervised image semantic segmentation results.

Work in progress

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Rethinking Atrous Convolution for Semantic Image Segmentation
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Energy-based Generative Adversarial Network
• High-Resolution Representations for Labeling Pixels and Regions
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations
• Maximum Entropy Generators for Energy-Based Models
• Generating Multiple Objects at Spatially Distinct Locations