Full-Network Embedding in a Multimodal Embedding Pipeline

Abstract

The current state-of-the-art for image annotation and image retrieval tasks is obtained through deep neural networks, which combine an image representation and a text representation into a shared embedding space. In this paper we evaluate the impact of using the Full-Network embedding in this setting, replacing the original image representation in a competitive multimodal embedding generation scheme. Unlike the one-layer image embeddings typically used by most approaches, the Full-Network embedding provides a multi-scale representation of images, which results in richer characterizations. To measure the influence of the Full-Network embedding, we evaluate its performance on three different datasets, and compare the results with the original multimodal embedding generation scheme when using a one-layer image embedding, and with the rest of the state-of-the-art. Results for image annotation and image retrieval tasks indicate that the Full-Network embedding is consistently superior to the one-layer embedding. These results motivate the integration of the Full-Network embedding on any multimodal embedding generation scheme, something feasible thanks to the flexibility of the approach.

In 2nd Workshop on Semantic Deep Learning (SemDeep-2) at the 12th International Conference on Computational Semantics (IWCS) 2017

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• Multimodal Convolutional Neural Networks for Matching Image and Sentence
• Word2VisualVec: Image and Video to Sentence Matching by Visual Feature Prediction
• Deep Convolutional Features for Image Based Retrieval and Scene Categorization
• Image Captioning with Deep Bidirectional LSTMs
• An Out-of-the-box Full-network Embedding for Convolutional Neural Networks
• On the Behavior of Convolutional Nets for Feature Extraction