RTIC: Residual Learning for Text and Image Composition using Graph Convolutional Network
arXiv:2104.03015
Abstract
In this paper, we study the compositional learning of images and texts for image retrieval. The query is given in the form of an image and text that describes the desired modifications to the image; the goal is to retrieve the target image that satisfies the given modifications and resembles the query by composing information in both the text and image modalities. To remedy this, we propose a novel architecture designed for the image-text composition task and show that the proposed structure can effectively encode the differences between the source and target images conditioned on the text. Furthermore, we introduce a new joint training technique based on the graph convolutional network that is generally applicable for any existing composition methods in a plug-and-play manner. We found that the proposed technique consistently improves performance and achieves state-of-the-art scores on various benchmarks. To avoid misleading experimental results caused by trivial training hyper-parameters, we reproduce all individual baselines and train models with a unified training environment. We expect this approach to suppress undesirable effects from irrelevant components and emphasize the image-text composition module's ability. Also, we achieve the state-of-the-art score without restricting the training environment, which implies the superiority of our method considering the gains from hyper-parameter tuning. The code, including all the baseline methods, are released https://github.com/nashory/rtic-gcn-pytorch.
References in corpus (14)
- Semi-Supervised Classification with Graph Convolutional Networks
- Practical Bayesian Optimization of Machine Learning Algorithms
- In Defense of the Triplet Loss for Person Re-Identification
- Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Multimodal Residual Learning for Visual QA
- Dialog-based Interactive Image Retrieval
- Modality-Agnostic Attention Fusion for visual search with text feedback
- Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback
- CurlingNet: Compositional Learning between Images and Text for Fashion IQ Data
- Semi-supervised Feature-Level Attribute Manipulation for Fashion Image Retrieval
- Fashion-IQ 2020 Challenge 2nd Place Team's Solution
- Neural Naturalist: Generating Fine-Grained Image Comparisons
- Cycled Compositional Learning between Images and Text