Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language

Abstract

Neuro-symbolic representations have proved effective in learning structure information in vision and language. In this paper, we propose a new model architecture for learning multi-modal neuro-symbolic representations for video captioning. Our approach uses a dictionary learning-based method of learning relations between videos and their paired text descriptions. We refer to these relations as relative roles and leverage them to make each token role-aware using attention. This results in a more structured and interpretable architecture that incorporates modality-specific inductive biases for the captioning task. Intuitively, the model is able to learn spatial, temporal, and cross-modal relations in a given pair of video and text. The disentanglement achieved by our proposal gives the model more capacity to capture multi-modal structures which result in captions with higher quality for videos. Our experiments on two established video captioning datasets verifies the effectiveness of the proposed approach based on automatic metrics. We further conduct a human evaluation to measure the grounding and relevance of the generated captions and observe consistent improvement for the proposed model. The codes and trained models can be found at https://github.com/hassanhub/R3Transformer

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Categorical Reparameterization with Gumbel-Softmax
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• The Kinetics Human Action Video Dataset
• Neural Discrete Representation Learning
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Learning Video Representations using Contrastive Bidirectional Transformer
• Towards Automatic Learning of Procedures from Web Instructional Videos
• RNNs Implicitly Implement Tensor Product Representations
• Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving
• Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators
• HUBERT Untangles BERT to Improve Transfer across NLP Tasks
• Mapping Natural-language Problems to Formal-language Solutions Using Structured Neural Representations
• MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
• Learning to Discretely Compose Reasoning Module Networks for Video Captioning
• Learning and analyzing vector encoding of symbolic representations