Best Prompts for Text-to-Image Models and How to Find Them

Abstract

Recent progress in generative models, especially in text-guided diffusion models, has enabled the production of aesthetically-pleasing imagery resembling the works of professional human artists. However, one has to carefully compose the textual description, called the prompt, and augment it with a set of clarifying keywords. Since aesthetics are challenging to evaluate computationally, human feedback is needed to determine the optimal prompt formulation and keyword combination. In this paper, we present a human-in-the-loop approach to learning the most useful combination of prompt keywords using a genetic algorithm. We also show how such an approach can improve the aesthetic appeal of images depicting the same descriptions.

13 pages (6 main pages), 7 figures, 4 tables, accepted at SIGIR '23 Short Paper Track

Cited by in corpus (7)

• PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
• A Prompt Log Analysis of Text-to-Image Generation Systems
• PromptMap: An Alternative Interaction Style for AI-Based Image Generation
• A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis
• Prompt Optimizer of Text-to-Image Diffusion Models for Abstract Concept Understanding
• Steering Large Text-to-Image Model for Abstract Art Synthesis: Preference-based Prompt Optimization and Visualization
• SSP: A Simple and Safe automatic Prompt engineering method towards realistic image synthesis on LVM