RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions

Abstract

Generative AI models have shown impressive ability to produce images with text prompts, which could benefit creativity in visual art creation and self-expression. However, it is unclear how precisely the generated images express contexts and emotions from the input texts. We explored the emotional expressiveness of AI-generated images and developed RePrompt, an automatic method to refine text prompts toward precise expression of the generated images. Inspired by crowdsourced editing strategies, we curated intuitive text features, such as the number and concreteness of nouns, and trained a proxy model to analyze the feature effects on the AI-generated image. With model explanations of the proxy model, we curated a rubric to adjust text prompts to optimize image generation for precise emotion expression. We conducted simulation and user studies, which showed that RePrompt significantly improves the emotional expressiveness of AI-generated images, especially for negative emotions.

arXiv commentTo appear in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23)

References in corpus (14)

• XGBoost: A Scalable Tree Boosting System
• Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
• Learning to Prompt for Vision-Language Models
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts
• DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models
• Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd Ideation
• Interpretable Directed Diversity: Leveraging Model Explanations for Iterative Crowd Ideation
• CLIPasso: Semantically-Aware Object Sketching
• SalienTrack: providing salient information for semi-automated self-tracking feedback with model explanations

Cited by in corpus (17)

• PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation
• PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
• PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
• Generative AI in the Wild: Prospects, Challenges, and Strategies
• ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models
• AIdeation: Designing a Human-AI Collaborative Ideation System for Concept Designers
• "When He Feels Cold, He Goes to the Seahorse"-Blending Generative AI into Multimaterial Storymaking for Family Expressive Arts Therapy
• A Preliminary Exploration of YouTubers' Use of Generative-AI in Content Creation
• Prompt Evolution for Generative AI: A Classifier-Guided Approach
• Designing for Human-Agent Alignment: Understanding what humans want from their agents
• From Fake Perfects to Conversational Imperfects: Exploring Image-Generative AI as a Boundary Object for Participatory Design of Public Spaces
• PromptMap: An Alternative Interaction Style for AI-Based Image Generation
• A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis
• GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Design
• "When I lost it, they dragged me out": How Care Encounters Empower Marginalized Young Adults' Aspiration and Mental Health Care-Seeking
• Varif.ai to Vary and Verify User-Driven Diversity in Scalable Image Generation
• Script2Screen: Supporting Dialogue Scriptwriting with Interactive Audiovisual Generation