Spatio-Temporal Graph for Video Captioning with Knowledge Distillation

Abstract

Video captioning is a challenging task that requires a deep understanding of visual scenes. State-of-the-art methods generate captions using either scene-level or object-level information but without explicitly modeling object interactions. Thus, they often fail to make visually grounded predictions, and are sensitive to spurious correlations. In this paper, we propose a novel spatio-temporal graph model for video captioning that exploits object interactions in space and time. Our model builds interpretable links and is able to provide explicit visual grounding. To avoid unstable performance caused by the variable number of objects, we further propose an object-aware knowledge distillation mechanism, in which local object information is used to regularize global scene features. We demonstrate the efficacy of our approach through extensive experiments on two benchmarks, showing our approach yields competitive performance with interpretable predictions.

arXiv commentCVPR 2020

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Semi-Supervised Classification with Graph Convolutional Networks
• The Kinetics Human Action Video Dataset
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Coherent Multi-Sentence Video Description with Variable Level of Detail
• Unifying distillation and privileged information
• SlowFast Networks for Video Recognition
• Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning

Cited by in corpus (5)

• Knowledge Distillation: A Survey
• iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering
• Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation
• STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training
• iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability