Data-efficient Alignment of Multimodal Sequences by Aligning Gradient Updates and Internal Feature Distributions

Abstract

The task of video and text sequence alignment is a prerequisite step toward joint understanding of movie videos and screenplays. However, supervised methods face the obstacle of limited realistic training data. With this paper, we attempt to enhance data efficiency of the end-to-end alignment network NeuMATCH [15]. Recent research [56] suggests that network components dealing with different modalities may overfit and generalize at different speeds, creating difficulties for training. We propose to employ (1) layer-wise adaptive rate scaling (LARS) to align the magnitudes of gradient updates in different layers and balance the pace of learning and (2) sequence-wise batch normalization (SBN) to align the internal feature distributions from different modalities. Finally, we leverage random projection to reduce the dimensionality of input features. On the YouTube Movie Summary dataset, the combined use of these technique closes the performance gap when the pretraining on the LSMDC dataset is omitted and achieves the state-of-the-art result. Extensive empirical comparisons and analysis reveal that these techniques improve optimization and regularize the network more effectively than two different setups of layer normalization.

arXiv commentThis paper is accepted to WACV2021

References in corpus (16)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Large Batch Training of Convolutional Networks
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• VideoBERT: A Joint Model for Video and Language Representation Learning
• BSN: Boundary Sensitive Network for Temporal Action Proposal Generation
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Visually Grounded Neural Syntax Acquisition
• Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
• TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
• End-to-End Dense Video Captioning with Masked Transformer
• Reconstruction Network for Video Captioning
• Differentiable Game Mechanics
• What Makes Training Multi-Modal Classification Networks Hard?
• Analogical Reasoning for Visually Grounded Language Acquisition
• A Quantitative Analysis of the Effect of Batch Normalization on Gradient Descent