PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement

Abstract

The recent advancements in Generative AI have significantly advanced the field of text-to-image generation. The state-of-the-art text-to-image model, Stable Diffusion, is now capable of synthesizing high-quality images with a strong sense of aesthetics. Crafting text prompts that align with the model's interpretation and the user's intent thus becomes crucial. However, prompting remains challenging for novice users due to the complexity of the stable diffusion model and the non-trivial efforts required for iteratively editing and refining the text prompts. To address these challenges, we propose PromptCharm, a mixed-initiative system that facilitates text-to-image creation through multi-modal prompt engineering and refinement. To assist novice users in prompting, PromptCharm first automatically refines and optimizes the user's initial prompt. Furthermore, PromptCharm supports the user in exploring and selecting different image styles within a large database. To assist users in effectively refining their prompts and images, PromptCharm renders model explanations by visualizing the model's attention values. If the user notices any unsatisfactory areas in the generated images, they can further refine the images through model attention adjustment or image inpainting within the rich feedback loop of PromptCharm. To evaluate the effectiveness and usability of PromptCharm, we conducted a controlled user study with 12 participants and an exploratory user study with another 12 participants. These two studies show that participants using PromptCharm were able to create images with higher quality and better aligned with the user's expectations compared with using two variants of PromptCharm that lacked interaction or visualization support.

arXiv commentTo appear in the 2024 CHI Conference on Human Factors in Computing Systems (CHI '24), May 11--16, 2024, Honolulu, HI, USA

References in corpus (13)

• Learning Transferable Visual Models From Natural Language Supervision
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Questioning the AI: Informing Design Practices for Explainable AI User Experiences
• Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
• Large-scale Text-to-Image Generation Models for Visual Artists' Creative Works
• PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation
• RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
• PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
• Best Prompts for Text-to-Image Models and How to Find Them
• Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
• GANzilla: User-Driven Direction Discovery in Generative Adversarial Networks
• GANravel: User-Driven Direction Disentanglement in Generative Adversarial Networks

Cited by in corpus (19)

• AIdeation: Designing a Human-AI Collaborative Ideation System for Concept Designers
• What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
• Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature
• AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools
• HarmonyCut: Supporting Creative Chinese Paper-cutting Design with Form and Connotation Harmony
• DesignFromX: Empowering Consumer-Driven Design Space Exploration through Feature Composition of Referenced Products
• Expandora: Broadening Design Exploration with Text-to-Image Model
• MemoVis: A GenAI-Powered Tool for Creating Companion Reference Images for 3D Design Feedback
• PromptMap: An Alternative Interaction Style for AI-Based Image Generation
• Exploring the Innovation Opportunities for Pre-trained Models
• Exploring Visual Prompts: Refining Images with Scribbles and Annotations in Generative AI Image Tools
• ThematicPlane: Bridging Tacit User Intent and Latent Spaces for Image Generation
• GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Design
• DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoning
• KinemaFX: A Kinematic-Driven Interactive System for Particle Effect Exploration and Customization
• Varif.ai to Vary and Verify User-Driven Diversity in Scalable Image Generation
• Empowering Children to Create AI-Enabled Augmented Reality Experiences
• Script2Screen: Supporting Dialogue Scriptwriting with Interactive Audiovisual Generation
• ESCT3D: Efficient and Selectively Controllable Text-Driven 3D Content Generation with Gaussian Splatting