Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval

Abstract

Vision-Language Pretraining (VLP) and Foundation models have been the go-to recipe for achieving SoTA performance on general benchmarks. However, leveraging these powerful techniques for more complex vision-language tasks, such as cooking applications, with more structured input data, is still little investigated. In this work, we propose to leverage these techniques for structured-text based computational cuisine tasks. Our strategy, dubbed VLPCook, first transforms existing image-text pairs to image and structured-text pairs. This allows to pretrain our VLPCook model using VLP objectives adapted to the strutured data of the resulting datasets, then finetuning it on downstream computational cooking tasks. During finetuning, we also enrich the visual encoder, leveraging pretrained foundation models (e.g. CLIP) to provide local and global textual context. VLPCook outperforms current SoTA by a significant margin (+3.3 Recall@1 absolute improvement) on the task of Cross-Modal Food Retrieval on the large Recipe1M dataset. We conduct further experiments on VLP to validate their importance, especially on the Recipe1M+ dataset. Finally, we validate the generalization of the approach to other tasks (i.e, Food Recognition) and domains with structured text such as the Medical domain on the ROCO dataset. The code is available here: https://github.com/mshukor/VLPCook

Code: https://github.com/mshukor/VLPCook

References in corpus (19)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• Flamingo: a Visual Language Model for Few-Shot Learning
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• CoCa: Contrastive Captioners are Image-Text Foundation Models
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• Contrastive Learning of Medical Visual Representations from Paired Images and Text
• PaLI: A Jointly-Scaled Multilingual Language-Image Model
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• How Much Can CLIP Benefit Vision-and-Language Tasks?
• Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
• Language Models Can See: Plugging Visual Controls in Text Generation
• Learning Text-Image Joint Embedding for Efficient Cross-Modal Retrieval with Deep Feature Engineering
• Retrieval-Augmented Transformer for Image Captioning
• Learning Structural Representations for Recipe Generation and Food Retrieval
• MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning