VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
arXiv:2107.02681
Abstract
Since visual perception can give rich information beyond text descriptions for world understanding, there has been increasing interest in leveraging visual grounding for language learning. Recently, vokenization (Tan and Bansal, 2020) has attracted attention by using the predictions of a text-to-image retrieval model as labels for language model supervision. Despite its success, the method suffers from approximation error of using finite image labels and the lack of vocabulary diversity of a small image-text dataset. To overcome these limitations, we present VidLanKD, a video-language knowledge distillation method for improving language understanding. We train a multi-modal teacher model on a video-text dataset, and then transfer its knowledge to a student language model with a text dataset. To avoid approximation error, we propose to use different knowledge distillation objectives. In addition, the use of a large-scale video-text dataset helps learn diverse and richer vocabularies. In our experiments, VidLanKD achieves consistent improvements over text-only language models and vokenization models, on several downstream language understanding tasks including GLUE, SQuAD, and SWAG. We also demonstrate the improved world knowledge, physical reasoning, and temporal reasoning capabilities of our model by evaluating on the GLUE-diagnostics, PIQA, and TRACIE datasets. Lastly, we present comprehensive ablation studies as well as visualizations of the learned text-to-video grounding results of our teacher and student language models. Our code and models are available at: https://github.com/zinengtang/VidLanKD
NeurIPS 2021 (19 pages)
References in corpus (8)
- Distilling the Knowledge in a Neural Network
- Learning Transferable Visual Models From Natural Language Supervision
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Microsoft COCO Captions: Data Collection and Evaluation Server
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
- Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
- HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training