Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books

Abstract

Books are a rich source of both fine-grained information, how a character, an object or a scene looks like, as well as high-level semantics, what someone is thinking, feeling and how these states evolve through a story. This paper aims to align books to their movie releases in order to provide rich descriptive explanations for visual content that go semantically far beyond the captions available in current datasets. To align movies and books we exploit a neural sentence embedding that is trained in an unsupervised way from a large corpus of books, as well as a video-text neural embedding for computing similarities between movie clips and sentences in the book. We propose a context-aware CNN to combine information from multiple sources. We demonstrate good quantitative performance for movie/book alignment and show several qualitative examples that showcase the diversity of tasks our model can be used for.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Going Deeper with Convolutions
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Explain Images with Multimodal Recurrent Neural Networks
• Show and Tell: A Neural Image Caption Generator
• Efficient Structured Prediction with Latent Variables for General Graphical Models
• A Dataset for Movie Description
• Don't Just Listen, Use Your Imagination: Leveraging Visual Common Sense for Non-Visual Tasks

Cited by in corpus (140)

• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Fine-Tuning Language Models from Human Preferences
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• Learning Video Representations using Contrastive Bidirectional Transformer
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• FinBERT: Financial Sentiment Analysis with Pre-trained Language Models
• Multimodal Machine Learning: A Survey and Taxonomy
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• Analysing Lexical Semantic Change with Contextualised Word Representations
• Adversarial Feature Matching for Text Generation
• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• Supervised and Unsupervised Neural Approaches to Text Readability
• Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• Character-level and Multi-channel Convolutional Neural Networks for Large-scale Authorship Attribution
• Language Models are Open Knowledge Graphs
• KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
• Probabilistic Reasoning via Deep Learning: Neural Association Models
• Efficient Vector Representation for Documents through Corruption
• Rethinking Positional Encoding in Language Pre-training
• See, Hear, and Read: Deep Aligned Representations
• Linguistic Features for Readability Assessment
• Sample Efficient Text Summarization Using a Single Pre-Trained Transformer
• A Survey on Deep Learning Methods for Robot Vision
• Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs
• Efficient Adaptation of Pretrained Transformers for Abstractive Summarization
• Context is Key: Grammatical Error Detection with Contextual Word Representations
• TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
• Explaining First Impressions: Modeling, Recognizing, and Explaining Apparent Personality from Videos
• What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning
• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
• Improving Question Answering with External Knowledge
• TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
• CAiRE: An Empathetic Neural Chatbot
• Optimal Subarchitecture Extraction For BERT
• SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios
• Adversarial Text Generation Without Reinforcement Learning
• An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation
• From Trailers to Storylines: An Efficient Way to Learn from Movies
• SemStyle: Learning to Generate Stylised Image Captions using Unaligned Text
• What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
• Evolution of transfer learning in natural language processing
• VirtualHome: Simulating Household Activities via Programs
• Learning Generic Sentence Representations Using Convolutional Neural Networks
• Pre-Trained Models: Past, Present and Future
• Dirichlet Variational Autoencoder for Text Modeling
• Learning semantic sentence representations from visually grounded language without lexical knowledge
• Speech-Based Visual Question Answering
• Empathetic BERT2BERT Conversational Model: Learning Arabic Language Generation with Little Data
• BERT-based Ensembles for Modeling Disclosure and Support in Conversational Social Media Text
• Accenture at CheckThat! 2020: If you say so: Post-hoc fact-checking of claims using transformer-based models
• An End-to-End Approach to Natural Language Object Retrieval via Context-Aware Deep Reinforcement Learning
• MC-BERT: Efficient Language Pre-Training via a Meta Controller
• Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization
• Learning Sentence Representation with Guidance of Human Attention
• Incorporating Structured Commonsense Knowledge in Story Completion
• Real-Time Execution of Large-scale Language Models on Mobile
• An Empirical Evaluation of Visual Question Answering for Novel Objects
• Variance-reduced Language Pretraining via a Mask Proposal Network
• Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos
• LazyFormer: Self Attention with Lazy Update
• An Automated, End-to-End Framework for Modeling Attacks From Vulnerability Descriptions
• Combining Pre-trained Word Embeddings and Linguistic Features for Sequential Metaphor Identification
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
• Transformers with Competitive Ensembles of Independent Mechanisms
• WebRED: Effective Pretraining And Finetuning For Relation Extraction On The Web
• Support-Set Based Cross-Supervision for Video Grounding
• Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling
• Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
• Knowledge Injection into Dialogue Generation via Language Models
• Back-Translated Task Adaptive Pretraining: Improving Accuracy and Robustness on Text Classification
• Kungfupanda at SemEval-2020 Task 12: BERT-Based Multi-Task Learning for Offensive Language Detection
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries
• Understanding the Downstream Instability of Word Embeddings
• Fine-tuning of Pre-trained Transformers for Hate, Offensive, and Profane Content Detection in English and Marathi
• Sentiment analysis in tweets: an assessment study from classical to modern text representation models
• Representing Sentences as Low-Rank Subspaces
• Eigen Analysis of Self-Attention and its Reconstruction from Partial Computation
• Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning
• Denoising based Sequence-to-Sequence Pre-training for Text Generation
• Pay Attention when Required
• Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models
• Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation
• Fake Sentence Detection as a Training Task for Sentence Encoding
• LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation
• Pun Generation with Surprise
• How Vulnerable Are Automatic Fake News Detection Methods to Adversarial Attacks?
• Explicit Pairwise Word Interaction Modeling Improves Pretrained Transformers for English Semantic Similarity Tasks
• Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model
• Large-scale Cloze Test Dataset Created by Teachers
• Undivided Attention: Are Intermediate Layers Necessary for BERT?
• Automatic learner summary assessment for reading comprehension
• Unsupervised Stylish Image Description Generation via Domain Layer Norm
• Bilingual Language Modeling, A transfer learning technique for Roman Urdu
• Mixed Membership Word Embeddings for Computational Social Science
• Shatter: An Efficient Transformer Encoder with Single-Headed Self-Attention and Relative Sequence Partitioning
• WikiCheck: An end-to-end open source Automatic Fact-Checking API based on Wikipedia
• Robustness of on-device Models: Adversarial Attack to Deep Learning Models on Android Apps
• Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task
• Improving latent variable descriptiveness with AutoGen
• Story-oriented Image Selection and Placement
• Semantic Text-to-Face GAN -ST^2FG
• Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference
• A Face-to-Face Neural Conversation Model
• Towards Robust Pattern Recognition: A Review
• Text Modeling with Syntax-Aware Variational Autoencoders
• Learning Joint Representations of Videos and Sentences with Web Image Search
• PARADE: A New Dataset for Paraphrase Identification Requiring Computer Science Domain Knowledge
• Recognising Biomedical Names: Challenges and Solutions
• How Furiously Can Colourless Green Ideas Sleep? Sentence Acceptability in Context
• LV-BERT: Exploiting Layer Variety for BERT
• Predicting In-game Actions from Interviews of NBA Players
• A Temporal Variational Model for Story Generation
• Contrastive Learning of Visual-Semantic Embeddings
• Data-efficient Alignment of Multimodal Sequences by Aligning Gradient Updates and Internal Feature Distributions
• Evaluating Document Coherence Modelling
• Semantic Frame Forecast
• CALM: Continuous Adaptive Learning for Language Modeling
• Evidence-Aware Inferential Text Generation with Vector Quantised Variational AutoEncoder
• Improving Distributed Representations of Tweets - Present and Future
• mAnI: Movie Amalgamation using Neural Imitation
• Weakly Supervised Learning of Heterogeneous Concepts in Videos
• Have Attention Heads in BERT Learned Constituency Grammar?