Deep Fragment Embeddings for Bidirectional Image Sentence Mapping

Abstract

We introduce a model for bidirectional retrieval of images and sentences through a multi-modal embedding of visual and natural language data. Unlike previous models that directly map images or sentences into a common embedding space, our model works on a finer level and embeds fragments of images (objects) and fragments of sentences (typed dependency tree relations) into a common space. In addition to a ranking objective seen in previous work, this allows us to add a new fragment alignment objective that learns to directly associate these fragments across modalities. Extensive experimental evaluation shows that reasoning on both the global level of images and sentences and the finer level of their respective fragments significantly improves performance on image-sentence retrieval tasks. Additionally, our model provides interpretable predictions since the inferred inter-modal fragment alignment is explicit.

References in corpus (1)

• ImageNet Large Scale Visual Recognition Challenge

Cited by in corpus (149)

• Learning Transferable Visual Models From Natural Language Supervision
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Text Understanding from Scratch
• Exploring Models and Data for Image Question Answering
• Deep Learning Techniques for Future Intelligent Cross-Media Retrieval
• L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
• Exploring Nearest Neighbor Approaches for Image Captioning
• MDMMT: Multidomain Multimodal Transformer for Video Retrieval
• Understanding Image Virality
• From Captions to Visual Concepts and Back
• Deep Image Retrieval: Learning global representations for image search
• Representation Learning for Natural Language Processing
• Cross-Modal Learning via Pairwise Constraints
• Recurrent Topic-Transition GAN for Visual Paragraph Generation
• Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
• COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
• Guiding Long-Short Term Memory for Image Caption Generation
• Detecting Visual Relationships with Deep Relational Networks
• Neural Attention for Image Captioning: Review of Outstanding Methods
• Multi-modal Deep Analysis for Multimedia
• Multimodal Convolutional Neural Networks for Matching Image and Sentence
• Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
• Learning Deep Structure-Preserving Image-Text Embeddings
• Stacked Cross Attention for Image-Text Matching
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training
• Interleaved Text/Image Deep Mining on a Large-Scale Radiology Database for Automated Image Interpretation
• Dual Attention Networks for Multimodal Reasoning and Matching
• Visual Question Answering: A Survey of Methods and Datasets
• Visual7W: Grounded Question Answering in Images
• Reconstruction Network for Video Captioning
• Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques
• Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning
• Research Status of Deep Learning Methods for Rumor Detection
• Towards a Visual Turing Challenge
• What value do explicit high level concepts have in vision to language problems?
• Multi-modal gated recurrent units for image description
• TediGAN: Text-Guided Diverse Face Image Generation and Manipulation
• Multimodal Transformer with Multi-View Visual Representation for Image Captioning
• IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval
• Image Captioning with Deep Bidirectional LSTMs
• Learning to Answer Questions From Image Using Convolutional Neural Network
• Natural Language Object Retrieval
• A Pooling Approach to Modelling Spatial Relations for Image Retrieval and Annotation
• Weakly-Supervised Alignment of Video With Text
• Learning Cross-Modal Deep Representations for Robust Pedestrian Detection
• Predicting Aircraft Trajectories: A Deep Generative Convolutional Recurrent Neural Networks Approach
• Multi-Cue Zero-Shot Learning with Strong Supervision
• Image Captioning and Visual Question Answering Based on Attributes and External Knowledge
• What can you do with a rock? Affordance extraction via word embeddings
• Consensus-Aware Visual-Semantic Embedding for Image-Text Matching
• Hierarchical LSTMs with Adaptive Attention for Visual Captioning
• Multimodal Grounding for Language Processing
• Solving Bongard Problems with a Visual Language and Pragmatic Reasoning
• Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition
• Learning to Globally Edit Images with Textual Description
• Saliency-Guided Attention Network for Image-Sentence Matching
• Automatic Concept Discovery from Parallel Text and Visual Corpora
• A Hierarchical Approach for Generating Descriptive Image Paragraphs
• Learning Cross-Modal Embeddings with Adversarial Networks for Cooking Recipes and Food Images
• Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts
• A Comprehensive Survey of Deep Learning for Image Captioning
• Towards Deconfounded Image-Text Matching with Causal Inference
• Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval
• A New Evaluation Protocol and Benchmarking Results for Extendable Cross-media Retrieval
• Learning Visual Features from Large Weakly Supervised Data
• Learning Structured Semantic Embeddings for Visual Recognition
• Deep Bayesian Active Learning for Multiple Correct Outputs
• TPsgtR: Neural-Symbolic Tensor Product Scene-Graph-Triplet Representation for Image Captioning
• Learning Universal Graph Neural Network Embeddings With Aid Of Transfer Learning
• Multilingual Visual Sentiment Concept Matching
• Deep Embedding for Spatial Role Labeling
• Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
• Stacked Cross-modal Feature Consolidation Attention Networks for Image Captioning
• LCEval: Learned Composite Metric for Caption Evaluation
• Improving Deep Visual Representation for Person Re-identification by Global and Local Image-language Association
• Dense Captioning with Joint Inference and Visual Context
• Instance-aware Image and Sentence Matching with Selective Multimodal LSTM
• Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task
• Graph Structured Network for Image-Text Matching
• Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation
• Deep Bayesian Active Learning, A Brief Survey on Recent Advances
• A Baseline for the Commands For Autonomous Vehicles Challenge
• Analyzing Classifiers: Fisher Vectors and Deep Neural Networks
• Recurrent Neural Network-Based Semantic Variational Autoencoder for Sequence-to-Sequence Learning
• Integrating Physiological Time Series and Clinical Notes with Deep Learning for Improved ICU Mortality Prediction
• Deep Binaries: Encoding Semantic-Rich Cues for Efficient Textual-Visual Cross Retrieval
• GXNOR-Net: Training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework
• Learning the semantic structure of objects from Web supervision
• Probabilistic Embeddings for Cross-Modal Retrieval
• Evaluation of Audio-Visual Alignments in Visually Grounded Speech Models
• Visual Grounding of Whole Radiology Reports for 3D CT Images
• Reconstruct and Represent Video Contents for Captioning via Reinforcement Learning
• Image Captioning with Context-Aware Auxiliary Guidance
• Full-Network Embedding in a Multimodal Embedding Pipeline
• Multimodal Differential Network for Visual Question Generation
• Gaussian Smoothen Semantic Features (GSSF) -- Exploring the Linguistic Aspects of Visual Captioning in Indian Languages (Bengali) Using MSCOCO Framework
• Joint Visual-Textual Embedding for Multimodal Style Search
• Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences
• Revisiting Cross Modal Retrieval
• Discover and Learn New Objects from Documentaries
• Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-modal Retrieval
• Phrase Localization Without Paired Training Examples
• Rethinking the Artificial Neural Networks: A Mesh of Subnets with a Central Mechanism for Storing and Predicting the Data
• Joint Multimedia Event Extraction from Video and Article
• Giving Commands to a Self-driving Car: A Multimodal Reasoner for Visual Grounding
• A Joint Network for Grasp Detection Conditioned on Natural Language Commands
• A Deep Decoder Structure Based on WordEmbedding Regression for An Encoder-Decoder Based Model for Image Captioning
• Improving Image Captioning by Leveraging Knowledge Graphs
• Relation-aware Instance Refinement for Weakly Supervised Visual Grounding
• TIGEr: Text-to-Image Grounding for Image Caption Evaluation
• Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval
• Multitask Text-to-Visual Embedding with Titles and Clickthrough Data
• Linguistic Structures as Weak Supervision for Visual Scene Graph Generation
• Hierarchical Photo-Scene Encoder for Album Storytelling
• A Procedural Texture Generation Framework Based on Semantic Descriptions
• Multi-Modal Coreference Resolution with the Correlation between Space Structures
• Semi Supervised Phrase Localization in a Bidirectional Caption-Image Retrieval Framework
• Semantic Text-to-Face GAN -ST^2FG
• Beyond the Deep Metric Learning: Enhance the Cross-Modal Matching with Adversarial Discriminative Domain Regularization
• An Unsupervised Sampling Approach for Image-Sentence Matching Using Document-Level Structural Information
• Simple to Complex Cross-modal Learning to Rank
• Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries
• Self-view Grounding Given a Narrated 360° Video
• Deep Exemplar Networks for VQA and VQG
• Segment-Phrase Table for Semantic Segmentation, Visual Entailment and Paraphrasing
• Neural Twins Talk
• Visual Relation Grounding in Videos
• Multi-modal Transformer for Video Retrieval
• Semi-supervised Visual Feature Integration for Pre-trained Language Models
• Probability Trajectory: One New Movement Description for Trajectory Prediction
• Improving Clinical Outcome Predictions Using Convolution over Medical Entities with Multimodal Learning
• A Weighted Multi-Criteria Decision Making Approach for Image Captioning
• Watch and Learn: Mapping Language and Noisy Real-world Videos with Self-supervision
• Reconstructing and grounding narrated instructional videos in 3D
• Neural Twins Talk & Alternative Calculations
• Paraphrase Thought: Sentence Embedding Module Imitating Human Language Recognition
• Object-sensitive Deep Reinforcement Learning
• Target-Tailored Source-Transformation for Scene Graph Generation
• Survey of Visual-Semantic Embedding Methods for Zero-Shot Image Retrieval
• Joint Learning of Distributed Representations for Images and Texts
• Phrase Grounding by Soft-Label Chain Conditional Random Field
• Improving Visual Recognition using Ambient Sound for Supervision