Text-to-Image Generation Grounded by Fine-Grained User Attention
arXiv:2011.03775
Abstract
Localized Narratives is a dataset with detailed natural language descriptions of images paired with mouse traces that provide a sparse, fine-grained visual grounding for phrases. We propose TReCS, a sequential model that exploits this grounding to generate images. TReCS uses descriptions to retrieve segmentation masks and predict object labels aligned with mouse traces. These alignments are used to select and position masks to generate a fully covered segmentation canvas; the final image is produced by a segmentation-to-image generator using this canvas. This multi-step, retrieval-based approach outperforms existing direct text-to-image generation models on both automatic metrics and human evaluations: overall, its generated images are more photo-realistic and better match descriptions.
arXiv commentTo appear in WACV 2021
References in corpus (8)
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Generative Adversarial Text to Image Synthesis
- A Neural Representation of Sketch Drawings
- The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
- A Note on the Inception Score
- Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis
- End-to-End Retrieval in Continuous Space
- Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO