Deep Consensus Learning

Abstract

Both generative learning and discriminative learning have recently witnessed remarkable progress using Deep Neural Networks (DNNs). For structured input synthesis and structured output prediction problems (e.g., layout-to-image synthesis and image semantic segmentation respectively), they often are studied separately. This paper proposes deep consensus learning (DCL) for joint layout-to-image synthesis and weakly-supervised image semantic segmentation. The former is realized by a recently proposed LostGAN approach, and the latter by introducing an inference network as the third player joining the two-player game of LostGAN. Two deep consensus mappings are exploited to facilitate training the three networks end-to-end: Given an input layout (a list of object bounding boxes), the generator generates a mask (label map) and then use it to help synthesize an image. The inference network infers the mask for the synthesized image. Then, the latent consensus is measured between the mask generated by the generator and the one inferred by the inference network. For the real image corresponding to the input layout, its mask also is computed by the inference network, and then used by the generator to reconstruct the real image. Then, the data consensus is measured between the real image and its reconstructed image. The discriminator still plays the role of an adversary by computing the realness scores for a real image, its reconstructed image and a synthesized image. In experiments, our DCL is tested in the COCO-Stuff dataset. It obtains compelling layout-to-image synthesis results and weakly-supervised image semantic segmentation results.

arXiv commentWork in progress

References in corpus (29)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• Deep Residual Learning for Image Recognition
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Improved Techniques for Training GANs
• Energy-based Generative Adversarial Network
• Fully Convolutional Networks for Semantic Segmentation
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• High-Resolution Representations for Labeling Pixels and Regions
• Stacked Hourglass Networks for Human Pose Estimation
• Pyramid Scene Parsing Network
• DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
• Semantic Image Synthesis with Spatially-Adaptive Normalization
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Hierarchical Implicit Models and Likelihood-Free Variational Inference
• Deep Directed Generative Models with Energy-Based Probability Estimation
• Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations
• Maximum Entropy Generators for Energy-Based Models
• Image Generation from Scene Graphs
• Generating Multiple Objects at Spatially Distinct Locations
• Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis
• Convolutional Simplex Projection Network (CSPN) for Weakly Supervised Semantic Segmentation