Inducing Hierarchical Compositional Model by Sparsifying Generator Network

Abstract

This paper proposes to learn hierarchical compositional AND-OR model for interpretable image synthesis by sparsifying the generator network. The proposed method adopts the scene-objects-parts-subparts-primitives hierarchy in image representation. A scene has different types (i.e., OR) each of which consists of a number of objects (i.e., AND). This can be recursively formulated across the scene-objects-parts-subparts hierarchy and is terminated at the primitive level (e.g., wavelets-like basis). To realize this AND-OR hierarchy in image synthesis, we learn a generator network that consists of the following two components: (i) Each layer of the hierarchy is represented by an over-complete set of convolutional basis functions. Off-the-shelf convolutional neural architectures are exploited to implement the hierarchy. (ii) Sparsity-inducing constraints are introduced in end-to-end training, which induces a sparsely activated and sparsely connected AND-OR model from the initially densely connected generator network. A straightforward sparsity-inducing constraint is utilized, that is to only allow the top-$k$ basis functions to be activated at each layer (where $k$ is a hyper-parameter). The learned basis functions are also capable of image reconstruction to explain the input images. In experiments, the proposed method is tested on four benchmark datasets. The results show that meaningful and interpretable hierarchical representations are learned with better qualities of image synthesis and reconstruction obtained than baselines.

arXiv commentThis is the CVPR version

References in corpus (11)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Spike and slab variable selection: Frequentist and Bayesian strategies
• Adversarially Learned Inference
• Deep Learning of Part-based Representation of Data Using Sparse Autoencoders with Nonnegativity Constraints
• GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
• Winner-Take-All Autoencoders
• k-Sparse Autoencoders
• Are GANs Created Equal? A Large-Scale Study
• Alternating Back-Propagation for Generator Network