Image Captions are Natural Prompts for Text-to-Image Models

Abstract

With the rapid development of Artificial Intelligence Generated Content (AIGC), it has become a common practice to train models on synthetic data due to data-scarcity and privacy leakage problems. Owing to massive and diverse information conveyed in real images, it is challenging for text-to-image generative models to synthesize informative training data with hand-crafted prompts. Considering the impressive ability of large generative models, could such models directly synthesize good training images for prediction tasks with proper prompts? We offer an affirmative response to this question by proposing a simple yet effective method, validated through ImageNet classification. Specifically, we caption each real image with the advanced captioning model to obtain informative and faithful prompts that extract class-relevant information and clarify the polysemy of class names. The image captions and class names are concatenated to prompt generative models for training image synthesis. We show that this simple caption incorporation significantly boosts the informativeness of synthetic data therefore enhancing downstream model generalization. More importantly, besides improvements in data augmentation and privacy preservation, our experiments demonstrate that synthesized images can exceed real data in terms of out-of-distribution robustness.

31 pages, 2 figure, 15 tables. Codes are available at https://github.com/LeavesLei/Caption_in_Prompt

References in corpus (21)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Overcoming catastrophic forgetting in neural networks
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• LAION-5B: An open large-scale dataset for training next generation image-text models
• GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Improved Denoising Diffusion Probabilistic Models
• Learning Robust Global Representations by Penalizing Local Predictive Power
• Neural Photo Editing with Introspective Adversarial Networks
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Classification Accuracy Score for Conditional Generative Models
• Effective Data Augmentation With Diffusion Models
• Synthetic Data from Diffusion Models Improves ImageNet Classification
• ResNet strikes back: An improved training procedure in timm
• StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners
• Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA
• Leaving Reality to Imagination: Robust Classification via Generated Datasets
• Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation
• Not Just Pretty Pictures: Toward Interventional Data Augmentation Using Text-to-Image Generators