PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
arXiv:2308.05184 · doi:10.1145/3586183.3606777
Abstract
While diffusion-based text-to-image (T2I) models provide a simple and powerful way to generate images, guiding this generation remains a challenge. For concepts that are difficult to describe through language, users may struggle to create prompts. Moreover, many of these models are built as end-to-end systems, lacking support for iterative shaping of the image. In response, we introduce PromptPaint, which combines T2I generation with interactions that model how we use colored paints. PromptPaint allows users to go beyond language to mix prompts that express challenging concepts. Just as we iteratively tune colors through layered placements of paint on a physical canvas, PromptPaint similarly allows users to apply different prompts to different canvas areas and times of the generative process. Through a set of studies, we characterize different approaches for mixing prompts, design trade-offs, and socio-technical challenges for generative models. With PromptPaint we provide insight into future steerable generative tools.
arXiv commentAccepted to UIST2023
References in corpus (34)
- Denoising Diffusion Probabilistic Models
- Hierarchical Text-Conditional Image Generation with CLIP Latents
- Diffusion Models Beat GANs on Image Synthesis
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
- Evaluating Large Language Models Trained on Code
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
- Blended Diffusion for Text-driven Editing of Natural Images
- A Learned Representation For Artistic Style
- Classifier-Free Diffusion Guidance
- DreamFusion: Text-to-3D using 2D Diffusion
- Prompt-to-Prompt Image Editing with Cross Attention Control
- Make-A-Video: Text-to-Video Generation without Text-Video Data
- SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
- Adding Conditional Control to Text-to-Image Diffusion Models
- MusicLM: Generating Music From Text
- Swapping Autoencoder for Deep Image Manipulation
- RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
- Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise
- DiffEdit: Diffusion-based semantic image editing with mask guidance
- Sequential Gallery for Interactive Visual Design Optimization
- Phenaki: Variable Length Video Generation From Open Domain Textual Description
- Composer: Creative and Controllable Image Synthesis with Composable Conditions
- InstructPix2Pix: Learning to Follow Image Editing Instructions
- GANSlider: How Users Control Generative Models for Images using Multiple Sliders with and without Feedforward Information
- AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts
- Imagic: Text-Based Real Image Editing with Diffusion Models
- VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance
- Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models
- MagicMix: Semantic Mixing with Diffusion Models
- UniTune: Text-Driven Image Editing by Fine Tuning a Diffusion Model on a Single Image
- Compositional Visual Generation with Composable Diffusion Models
- GLIGEN: Open-Set Grounded Text-to-Image Generation
- Zero-shot Image-to-Image Translation
- HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing
Cited by in corpus (19)
- Homogenization Effects of Large Language Models on Human Creative Ideation
- PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
- DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models
- TypeDance: Creating Semantic Typographic Logos from Image through Personalized Generation
- Patchview: LLM-Powered Worldbuilding with Generative Dust and Magnet Visualization
- AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools
- DesignWeaver: Dimensional Scaffolding for Text-to-Image Product Design
- Misty: UI Prototyping Through Interactive Conceptual Blending
- HarmonyCut: Supporting Creative Chinese Paper-cutting Design with Form and Connotation Harmony
- DesignFromX: Empowering Consumer-Driven Design Space Exploration through Feature Composition of Referenced Products
- Textoshop: Interactions Inspired by Drawing Software to Facilitate Text Editing
- BlendScape: Enabling End-User Customization of Video-Conferencing Environments through Generative AI
- Advancing GUI for Generative AI: Charting the Design Space of Human-AI Interactions through Task Creativity and Complexity
- Intent Tagging: Exploring Micro-Prompting Interactions for Supporting Granular Human-GenAI Co-Creation Workflows
- StoryEnsemble: Enabling Dynamic Exploration & Iteration in the Design Process with AI and Forward-Backward Propagation
- Exploring Visual Prompts: Refining Images with Scribbles and Annotations in Generative AI Image Tools
- GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Design
- KinemaFX: A Kinematic-Driven Interactive System for Particle Effect Exploration and Customization
- Varif.ai to Vary and Verify User-Driven Diversity in Scalable Image Generation