MCEN: Bridging Cross-Modal Gap between Cooking Recipes and Dish Images with Latent Variable Model
arXiv:2004.01095
Abstract
Nowadays, driven by the increasing concern on diet and health, food computing has attracted enormous attention from both industry and research community. One of the most popular research topics in this domain is Food Retrieval, due to its profound influence on health-oriented applications. In this paper, we focus on the task of cross-modal retrieval between food images and cooking recipes. We present Modality-Consistent Embedding Network (MCEN) that learns modality-invariant representations by projecting images and texts to the same embedding space. To capture the latent alignments between modalities, we incorporate stochastic latent variables to explicitly exploit the interactions between textual and visual features. Importantly, our method learns the cross-modal alignments during training but computes embeddings of different modalities independently at inference time for the sake of efficiency. Extensive experimental results clearly demonstrate that the proposed MCEN outperforms all existing approaches on the benchmark Recipe1M dataset and requires less computational cost.
arXiv commentAccepted to CVPR 2020
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
- Diagnosing and Enhancing VAE Models
- Finding beans in burgers: Deep semantic-visual embedding with localization
- Learning Cross-Modal Embeddings with Adversarial Networks for Cooking Recipes and Food Images