Context-Aware Embeddings for Automatic Art Analysis

Abstract

Automatic art analysis aims to classify and retrieve artistic representations from a collection of images by using computer vision and machine learning techniques. In this work, we propose to enhance visual representations from neural networks with contextual artistic information. Whereas visual representations are able to capture information about the content and the style of an artwork, our proposed context-aware embeddings additionally encode relationships between different artistic attributes, such as author, school, or historical period. We design two different approaches for using context in automatic art analysis. In the first one, contextual data is obtained through a multi-task learning model, in which several attributes are trained together to find visual relationships between elements. In the second approach, context is obtained through an art-specific knowledge graph, which encodes relationships between artistic attributes. An exhaustive evaluation of both of our models in several art analysis problems, such as author identification, type classification, or cross-modal retrieval, show that performance is improved by up to 7.3% in art classification and 37.24% in retrieval when context-aware embeddings are used.

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• An Overview of Multi-Task Learning in Deep Neural Networks
• Graph Embedding Techniques, Applications, and Performance: A Survey
• Multi-Task Learning as Multi-Objective Optimization
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Recognizing Image Style
• MOON: A Mixed Objective Optimization Network for the Recognition of Facial Attributes
• Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature

Cited by in corpus (9)

• Not Only Generative Art: Stable Diffusion for Content-Style Disentanglement in Art Analysis
• Graph Neural Networks for Knowledge Enhanced Visual Representation of Paintings
• Multiple instance learning on deep features for weakly supervised object detection with extreme domain shifts
• Disembodied Machine Learning: On the Illusion of Objectivity in NLP
• A Dataset and Baselines for Visual Question Answering on Art
• Historical and Modern Features for Buddha Statue Classification
• Built Year Prediction from Buddha Face with Heterogeneous Labels
• GCNBoost: Artwork Classification by Label Propagation through a Knowledge Graph
• Understanding Art through Multi-Modal Retrieval in Paintings