Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning

Abstract

Cross-modal recipe retrieval has recently gained substantial attention due to the importance of food in people's lives, as well as the availability of vast amounts of digital cooking recipes and food images to train machine learning models. In this work, we revisit existing approaches for cross-modal recipe retrieval and propose a simplified end-to-end model based on well established and high performing encoders for text and images. We introduce a hierarchical recipe Transformer which attentively encodes individual recipe components (titles, ingredients and instructions). Further, we propose a self-supervised loss function computed on top of pairs of individual recipe components, which is able to leverage semantic relationships within recipes, and enables training using both image-recipe and recipe-only samples. We conduct a thorough analysis and ablation studies to validate our design choices. As a result, our proposed method achieves state-of-the-art performance in the cross-modal recipe retrieval task on the Recipe1M dataset. We make code and models publicly available.

arXiv commentCVPR 2021

References in corpus (12)

• Efficient Estimation of Word Representations in Vector Space
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Simple Framework for Contrastive Learning of Visual Representations
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Skip-Thought Vectors
• ChineseFoodNet: A large-scale Image Dataset for Chinese Food Recognition
• Learning Video Representations from Textual Web Supervision
• Dividing and Conquering Cross-Modal Recipe Retrieval: from Nearest Neighbours Baselines to SoTA
• Is Saki #delicious? The Food Perception Gap on Instagram and Its Relation to Health
• RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes
• Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanism
• Picture-to-Amount (PITA): Predicting Relative Ingredient Amounts from Food Images

Cited by in corpus (1)

• Cross-Modal Retrieval and Synthesis (X-MRS): Closing the Modality Gap in Shared Representation Learning