CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders
arXiv:2106.14843
Abstract
This work presents CLIPDraw, an algorithm that synthesizes novel drawings based on natural language input. CLIPDraw does not require any training; rather a pre-trained CLIP language-image encoder is used as a metric for maximizing similarity between the given description and a generated drawing. Crucially, CLIPDraw operates over vector strokes rather than pixel images, a constraint that biases drawings towards simpler human-recognizable shapes. Results compare between CLIPDraw and other synthesis-through-optimization methods, as well as highlight various interesting behaviors of CLIPDraw, such as satisfying ambiguous text in multiple ways, reliably producing drawings in diverse artistic styles, and scaling from simple to complex visual representations as stroke count is increased. Code for experimenting with the method is available at: https://colab.research.google.com/github/kvfrans/clipdraw/blob/main/clipdraw.ipynb
References in corpus (7)
- Conditional Generative Adversarial Nets
- Learning Transferable Visual Models From Natural Language Supervision
- Neural Discrete Representation Learning
- Zero-Shot Text-to-Image Generation
- Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
- Adversarial Text-to-Image Synthesis: A Review
- Generative Art Using Neural Visual Grammars and Dual Encoders
Cited by in corpus (6)
- Differentiable Quality Diversity
- CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
- Zero-Shot Text-Guided Object Generation with Dream Fields
- Telling Creative Stories Using Generative Visual Aids
- Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model
- StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis