Illiterate DALL-E Learns to Compose
arXiv:2110.11405
Abstract
Although DALL-E has shown an impressive ability of composition-based systematic generalization in image generation, it requires the dataset of text-image pairs and the compositionality is provided by the text. In contrast, object-centric representation models like the Slot Attention model learn composable representations without the text prompt. However, unlike DALL-E its ability to systematically generalize for zero-shot generation is significantly limited. In this paper, we propose a simple but novel slot-based autoencoding architecture, called SLATE, for combining the best of both worlds: learning object-centric representations that allows systematic generalization in zero-shot image generation without text. As such, this model can also be seen as an illiterate DALL-E model. Unlike the pixel-mixture decoders of existing object-centric representation models, we propose to use the Image GPT decoder conditioned on the slots for capturing complex interactions among the slots and pixels. In experiments, we show that this simple and easy-to-implement architecture not requiring a text prompt achieves significant improvement in in-distribution and out-of-distribution (zero-shot) image generation and qualitatively comparable or better slot-attention structure than the models based on mixture decoders.
Published as a conference paper at ICLR 2022
References in corpus (13)
- Language Models are Few-Shot Learners
- Deep Convolutional Inverse Graphics Network
- Learning What and Where to Draw
- MONet: Unsupervised Scene Decomposition and Representation
- On the Binding Problem in Artificial Neural Networks
- Generating Images Part by Part with Composite Generative Adversarial Networks
- Using latent space regression to analyze and leverage compositionality in GANs
- SCALOR: Generative World Models with Scalable Object Representations
- Towards causal generative scene models via competition of experts
- GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
- SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
- Transformers with Competitive Ensembles of Independent Mechanisms
- Generative Video Transformer: Can Objects be the Words?