PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation
arXiv:2307.09036 · doi:10.1109/TVCG.2023.3327168
Abstract
Generative text-to-image models have gained great popularity among the public for their powerful capability to generate high-quality images based on natural language prompts. However, developing effective prompts for desired images can be challenging due to the complexity and ambiguity of natural language. This research proposes PromptMagician, a visual analysis system that helps users explore the image results and refine the input prompts. The backbone of our system is a prompt recommendation model that takes user prompts as input, retrieves similar prompt-image pairs from DiffusionDB, and identifies special (important and relevant) prompt keywords. To facilitate interactive prompt refinement, PromptMagician introduces a multi-level visualization for the cross-modal embedding of the retrieved images and recommended keywords, and supports users in specifying multiple criteria for personalized exploration. Two usage scenarios, a user study, and expert interviews demonstrate the effectiveness and usability of our system, suggesting it facilitates prompt engineering and improves the creativity support of the generative text-to-image model.
arXiv commentAccepted full paper for IEEE VIS 2023
References in corpus (19)
- Training language models to follow instructions with human feedback
- Hierarchical Text-Conditional Image Generation with CLIP Latents
- Imagen Video: High Definition Video Generation with Diffusion Models
- Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold
- A Taxonomy of Prompt Modifiers for Text-To-Image Generation
- RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
- EmoCo: Visual Analysis of Emotion Coherence in Presentation Videos
- CrossData: Leveraging Text-Data Connections for Authoring Data Documents
- Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains
- Optimizing Prompts for Text-to-Image Generation
- XNLI: Explaining and Diagnosing NLI-based Visual Data Analysis
- Explaining with Examples: Lessons Learned from Crowdsourced Introductory Description of Information Visualizations
- Anchorage: Visual Analysis of Satisfaction in Customer Service Videos via Anchor Events
- GestureLens: Visual Analysis of Gestures in Presentation Videos
- Computational Approaches for Traditional Chinese Painting: From the "Six Principles of Painting" Perspective
- DPVisCreator: Incorporating Pattern Constraints to Privacy-preserving Visualizations via Differential Privacy
- VideoPro: A Visual Analytics Approach for Interactive Video Programming
- Beyond Interestingness: Semantic and Context-Aware Natural Language Query Recommendations for Visual Data Analysis
- Multi-level tree based approach for interactive graph visualization with semantic zoom
Cited by in corpus (14)
- PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
- KNOWNET: Guided Health Information Seeking from LLMs via Knowledge Graph Integration
- VideoPro: A Visual Analytics Approach for Interactive Video Programming
- Prompting for products: Investigating design space exploration strategies for text-to-image generative models
- Visualization for Trust in Machine Learning Revisited: The State of the Field in 2023
- MemoVis: A GenAI-Powered Tool for Creating Companion Reference Images for 3D Design Feedback
- PromptMap: An Alternative Interaction Style for AI-Based Image Generation
- JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Models
- SPROUT: an Interactive Authoring Tool for Generating Programming Tutorials with the Visualization of Large Language Models
- DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoning
- GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Design
- Interactive Discovery and Exploration of Visual Bias in Generative Text-to-Image Models
- Varif.ai to Vary and Verify User-Driven Diversity in Scalable Image Generation
- SSP: A Simple and Safe automatic Prompt engineering method towards realistic image synthesis on LVM