VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs

Abstract

We present \textsc{Vx2Text}, a framework for text generation from multimodal inputs consisting of video plus text, speech, or audio. In order to leverage transformer networks, which have been shown to be effective at modeling language, each modality is first converted into a set of language embeddings by a learnable tokenizer. This allows our approach to perform multimodal fusion in the language space, thus eliminating the need for ad-hoc cross-modal fusion modules. To address the non-differentiability of tokenization on continuous inputs (e.g., video or audio), we utilize a relaxation scheme that enables end-to-end training. Furthermore, unlike prior encoder-only models, our network includes an autoregressive decoder to generate open-ended text from the multimodal embeddings fused by the language encoder. This renders our approach fully generative and makes it directly applicable to different "video+$x$ to text" problems without the need to design specialized network heads for each task. The proposed framework is not only conceptually simple but also remarkably effective: experiments demonstrate that our approach based on a single architecture outperforms the state-of-the-art on three video-based text-generation tasks -- captioning, question answering and audio-visual scene-aware dialog.

arXiv commentWork in progress

References in corpus (16)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Categorical Reparameterization with Gumbel-Softmax
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Visual Madlibs: Fill in the blank Image Generation and Question Answering
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• TVQA: Localized, Compositional Video Question Answering
• Visual7W: Grounded Question Answering in Images
• TVQA+: Spatio-Temporal Grounding for Video Question Answering
• Audio Visual Scene-Aware Dialog (AVSD) Challenge at DSTC7
• Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement
• TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval
• OBJ2TEXT: Generating Visually Descriptive Language from Object Layouts
• TMT: A Transformer-based Modal Translator for Improving Multimodal Sequence Representations in Audio Visual Scene-aware Dialog