Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation

Abstract

Have you ever looked at a painting and wondered what is the story behind it? This work presents a framework to bring art closer to people by generating comprehensive descriptions of fine-art paintings. Generating informative descriptions for artworks, however, is extremely challenging, as it requires to 1) describe multiple aspects of the image such as its style, content, or composition, and 2) provide background and contextual knowledge about the artist, their influences, or the historical period. To address these challenges, we introduce a multi-topic and knowledgeable art description framework, which modules the generated sentences according to three artistic topics and, additionally, enhances each description with external knowledge. The framework is validated through an exhaustive analysis, both quantitative and qualitative, as well as a comparative human evaluation, demonstrating outstanding results in terms of both topic diversity and information veracity.

arXiv commentICCV 2021

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Deep Residual Learning for Image Recognition
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Evaluation of Text Generation: A Survey
• Hierarchical Neural Story Generation
• Toward Controlled Generation of Text
• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects