Semantic Object Accuracy for Generative Text-to-Image Synthesis

Abstract

Generative adversarial networks conditioned on textual image descriptions are capable of generating realistic-looking images. However, current methods still struggle to generate images based on complex image captions from a heterogeneous domain. Furthermore, quantitatively evaluating these text-to-image models is challenging, as most evaluation metrics only judge image quality but not the conformity between the image and its caption. To address these challenges we introduce a new model that explicitly models individual objects within an image and a new evaluation metric called Semantic Object Accuracy (SOA) that specifically evaluates images given an image caption. The SOA uses a pre-trained object detector to evaluate if a generated image contains objects that are mentioned in the image caption, e.g. whether an image generated from "a car driving down the street" contains a car. We perform a user study comparing several text-to-image models and show that our SOA metric ranks the models the same way as humans, whereas other metrics such as the Inception Score do not. Our evaluation also shows that models which explicitly model objects outperform models which only model global image characteristics.

Added a user study to verify results. Code available at https://github.com/tohinz/semantic-object-accuracy-for-generative-text-to-image-synthesis

References in corpus (21)

• Conditional Generative Adversarial Nets
• YOLOv3: An Incremental Improvement
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Generative Adversarial Text to Image Synthesis
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• A note on the evaluation of generative models
• A Note on the Inception Score
• Learning What and Where to Draw
• Classification Accuracy Score for Conditional Generative Models
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
• ChatPainter: Improving Text to Image Generation using Dialogue
• Controllable Text-to-Image Generation
• Learning Hierarchical Semantic Image Manipulation through Structured Representations
• Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language
• PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph
• LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators
• Interactive Image Generation Using Scene Graphs
• Semantically Invariant Text-to-Image Generation
• Realistic Image Generation using Region-phrase Attention

Cited by in corpus (23)

• Blended Diffusion for Text-driven Editing of Natural Images
• Adversarial Text-to-Image Synthesis: A Review
• SpaText: Spatio-Textual Representation for Controllable Image Generation
• LAFITE: Towards Language-Free Training for Text-to-Image Generation
• Unifying Multimodal Transformer for Bi-directional Image and Text Generation
• Cross-Modal Contrastive Learning for Text-to-Image Generation
• ALR-GAN: Adaptive Layout Refinement for Text-to-Image Synthesis
• ARMANI: Part-level Garment-Text Alignment for Unified Cross-Modal Fashion Design
• DSE-GAN: Dynamic Semantic Evolution Generative Adversarial Network for Text-to-Image Generation
• Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis
• Adma-GAN: Attribute-Driven Memory Augmented GANs for Text-to-Image Generation
• Leveraging Visual Question Answering to Improve Text-to-Image Synthesis
• CPGAN: Full-Spectrum Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis
• When ControlNet Meets Inexplicit Masks: A Case Study of ControlNet on its Contour-following Ability
• Offline Evaluation of Set-Based Text-to-Image Generation
• GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks
• Attributes Aware Face Generation with Generative Adversarial Networks
• Deep Image Synthesis from Intuitive User Input: A Review and Perspectives
• Improving Text to Image Generation using Mode-seeking Function
• T2IW: Joint Text to Image & Watermark Generation
• RealCustom++: Representing Images as Real Textual Word for Real-Time Customization
• FA-GAN: Feature-Aware GAN for Text to Image Synthesis
• Using Text to Teach Image Retrieval