Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision

Abstract

Humans learn language by listening, speaking, writing, reading, and also, via interaction with the multimodal real world. Existing language pre-training frameworks show the effectiveness of text-only self-supervision while we explore the idea of a visually-supervised language model in this paper. We find that the main reason hindering this exploration is the large divergence in magnitude and distributions between the visually-grounded language datasets and pure-language corpora. Therefore, we develop a technique named "vokenization" that extrapolates multimodal alignments to language-only data by contextually mapping language tokens to their related images (which we call "vokens"). The "vokenizer" is trained on relatively small image captioning datasets and we then apply it to generate vokens for large language corpora. Trained with these contextually generated vokens, our visually-supervised language models show consistent improvements over self-supervised alternatives on multiple pure-language tasks such as GLUE, SQuAD, and SWAG. Code and pre-trained models publicly available at https://github.com/airsplay/vokenization

arXiv commentEMNLP 2020 (15 pages)

References in corpus (16)

• Distributed Representations of Words and Phrases and their Compositionality
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distributed Representations of Sentences and Documents
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• VQA: Visual Question Answering
• Skip-Thought Vectors
• Semi-supervised Sequence Learning
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• UNITER: UNiversal Image-TExt Representation Learning
• Grounded Language Learning in a Simulated 3D World
• Visually Grounded Neural Syntax Acquisition
• Contextual Word Representations: A Contextual Introduction
• Predicting Actions to Help Predict Translations

Cited by in corpus (3)

• Discrete and continuous representations and processing in deep learning: Looking forward
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer