OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation

Abstract

In this paper, we propose an Omni-perception Pre-Trainer (OPT) for cross-modal understanding and generation, by jointly modeling visual, text and audio resources. OPT is constructed in an encoder-decoder framework, including three single-modal encoders to generate token-based embeddings for each modality, a cross-modal encoder to encode the correlations among the three modalities, and two cross-modal decoders to generate text and image respectively. For the OPT's pre-training, we design a multi-task pretext learning scheme to model multi-modal resources from three different data granularities, \ie, token-, modality-, and sample-level modeling, through which OPT learns to align and translate among different modalities. The pre-training task is carried out on a large amount of image-text-audio triplets from Open Images. Experimental results show that OPT can learn strong image-text-audio multi-modal representations and achieve promising results on a variety of cross-modal understanding and generation tasks.

References in corpus (12)

• Bootstrap your own latent: A new approach to self-supervised Learning
• Generative Adversarial Text to Image Synthesis
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Zero-Shot Text-to-Image Generation
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Multi-Format Contrastive Learning of Audio Representations
• Improving speech emotion recognition via Transformer-based Predictive Coding through transfer learning
• Listen, Look and Deliberate: Visual context-aware speech recognition using pre-trained text-video representations

Cited by in corpus (3)

• VLP: A Survey on Vision-Language Pre-training
• VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
• Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation