Unsupervised Program Synthesis for Images By Sampling Without Replacement

Abstract

Program synthesis has emerged as a successful approach to the image parsing task. Most prior works rely on a two-step scheme involving supervised pretraining of a Seq2Seq model with synthetic programs followed by reinforcement learning (RL) for fine-tuning with real reference images. Fully unsupervised approaches promise to train the model directly on the target images without requiring curated pretraining datasets. However, they struggle with the inherent sparsity of meaningful programs in the search space. In this paper, we present the first unsupervised algorithm capable of parsing constructive solid geometry (CSG) images into context-free grammar (CFG) without pretraining via non-differentiable renderer. To tackle the \emph{non-Markovian} sparse reward problem, we combine three key ingredients -- (i) a grammar-encoded tree LSTM ensuring program validity (ii) entropy regularization and (iii) sampling without replacement from the CFG syntax tree. Empirically, our algorithm recovers meaningful programs in large search spaces (up to $3.8\times 10^{28}$). Further, even though our approach is fully unsupervised, it generalizes better than supervised methods on the synthetic 2D CSG dataset. On the 2D computer aided design (CAD) dataset, our approach significantly outperforms the supervised pretrained model and is competitive to the refined model.

Accepted to UAI 2021

References in corpus (12)

• Neural Discrete Representation Learning
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Disentangling by Factorising
• A Neural Representation of Sketch Drawings
• DeepCoder: Learning to Write Programs
• Synthesizing Programs for Images using Reinforced Adversarial Learning
• Neuro-Symbolic Program Synthesis
• Learning to Infer and Execute 3D Shape Programs
• Generative Code Modeling with Graphs
• UCSG-Net -- Unsupervised Discovering of Constructive Solid Geometry Tree
• Automatic Program Synthesis of Long Programs with a Learned Garbage Collector
• Synthetic Datasets for Neural Program Synthesis