Prompt Evolution for Generative AI: A Classifier-Guided Approach

Abstract

Synthesis of digital artifacts conditioned on user prompts has become an important paradigm facilitating an explosion of use cases with generative AI. However, such models often fail to connect the generated outputs and desired target concepts/preferences implied by the prompts. Current research addressing this limitation has largely focused on enhancing the prompts before output generation or improving the model's performance up front. In contrast, this paper conceptualizes prompt evolution, imparting evolutionary selection pressure and variation during the generative process to produce multiple outputs that satisfy the target concepts/preferences better. We propose a multi-objective instantiation of this broader idea that uses a multi-label image classifier-guided approach. The predicted labels from the classifiers serve as multiple objectives to optimize, with the aim of producing diversified images that meet user preferences. A novelty of our evolutionary algorithm is that the pre-trained generative model gives us implicit mutation operations, leveraging the model's stochastic generative capability to automate the creation of Pareto-optimized images more faithful to user preferences.

This work is published in the Proceedings of the IEEE Conference on Artificial Intelligence (CAI 2023). IEEE copyrights applies

References in corpus (9)

• Hierarchical Text-Conditional Image Generation with CLIP Latents
• eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers
• A Taxonomy of Prompt Modifiers for Text-To-Image Generation
• Evolutionary Multitasking for Multiobjective Continuous Optimization: Benchmark Problems, Performance Metrics and Baseline Results
• RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
• Fine-tuning language models to find agreement among humans with diverse preferences
• Re-Imagen: Retrieval-Augmented Text-to-Image Generator
• Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models
• Grimm in Wonderland: Prompt Engineering with Midjourney to Illustrate Fairytales

Cited by in corpus (2)

• Generative AI-based Prompt Evolution Engineering Design Optimization With Vision-Language Model
• LLM2TEA: An Agentic AI Designer for Discovery with Generative Evolutionary Multitasking