Conditional Set Generation with Transformers

Abstract

A set is an unordered collection of unique elements--and yet many machine learning models that generate sets impose an implicit or explicit ordering. Since model performance can depend on the choice of order, any particular ordering can lead to sub-optimal results. An alternative solution is to use a permutation-equivariant set generator, which does not specify an order-ing. An example of such a generator is the DeepSet Prediction Network (DSPN). We introduce the Transformer Set Prediction Network (TSPN), a flexible permutation-equivariant model for set prediction based on the transformer, that builds upon and outperforms DSPN in the quality of predicted set elements and in the accuracy of their predicted sizes. We test our model on MNIST-as-point-clouds (SET-MNIST) for point-cloud generation and on CLEVR for object detection.

6 pages, 6 figures, ICML 2020 Workshop on Object-Oriented Learning

References in corpus (6)

• Learning Representations and Generative Models for 3D Point Clouds
• Normalizing Flows for Probabilistic Modeling and Inference
• Maximum Weight Matching via Max-Product Belief Propagation
• FSPool: Learning Set Representations with Featurewise Sort Pooling
• Towards Differentiable Resampling
• Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks

Cited by in corpus (10)

• Object-Centric Learning with Slot Attention
• EPiC-GAN: Equivariant Point Cloud Generation for Particle Jets
• ATISS: Autoregressive Transformers for Indoor Scene Synthesis
• Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation
• Set-to-Sequence Methods in Machine Learning: a Review
• Set Representation Learning with Generalized Sliced-Wasserstein Embeddings
• Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation
• Top-N: Equivariant set and graph generation without exchangeability
• multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning
• Set Prediction without Imposing Structure as Conditional Density Estimation