An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

Abstract

Text-to-image models offer unprecedented freedom to guide creation through natural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them in new roles and novel scenes. In other words, we ask: how can we use language-guided models to turn our cat into a painting, or imagine a new product based on our favorite toy? Here we present a simple approach that allows such creative freedom. Using only 3-5 images of a user-provided concept, like an object or a style, we learn to represent it through new "words" in the embedding space of a frozen text-to-image model. These "words" can be composed into natural language sentences, guiding personalized creation in an intuitive way. Notably, we find evidence that a single word embedding is sufficient for capturing unique and varied concepts. We compare our approach to a wide range of baselines, and demonstrate that it can more faithfully portray the concepts across a range of applications and tasks. Our code, data and new words will be available at: https://textual-inversion.github.io

arXiv commentProject page: https://textual-inversion.github.io

Cited by in corpus (18)

• Large-scale Text-to-Image Generation Models for Visual Artists' Creative Works
• SpaText: Spatio-Textual Representation for Controllable Image Generation
• Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges
• DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics
• SGDiff: A Style Guided Diffusion Model for Fashion Synthesis
• DALLE-URBAN: Capturing the urban design expertise of large text to image transformers
• Single-Image 3D Human Digitization with Shape-Guided Diffusion
• The Infinite Index: Information Retrieval on Generative Text-To-Image Models
• From Paper to Card: Transforming Design Implications with Generative AI
• Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models
• CLIP4Sketch: Enhancing Sketch to Mugshot Matching through Dataset Augmentation using Diffusion Models
• Controllable Face Synthesis with Semantic Latent Diffusion Models
• PartEdit: Fine-Grained Image Editing using Pre-Trained Diffusion Models
• Conditional Distribution Modelling for Few-Shot Image Synthesis with Diffusion Models
• GEMRec: Towards Generative Model Recommendation
• One-shot Unsupervised Domain Adaptation with Personalized Diffusion Models
• Bridge Diffusion Model: Bridge Chinese Text-to-Image Diffusion Model with English Communities
• MTFusion: Reconstructing Any 3D Object from Single Image Using Multi-word Textual Inversion