OmniNet: A unified architecture for multi-modal multi-task learning

Abstract

Transformer is a popularly used neural network architecture, especially for language understanding. We introduce an extended and unified architecture that can be used for tasks involving a variety of modalities like image, text, videos, etc. We propose a spatio-temporal cache mechanism that enables learning spatial dimension of the input in addition to the hidden states corresponding to the temporal input sequence. The proposed architecture further enables a single model to support tasks with multiple input modalities as well as asynchronous multi-task learning, thus we refer to it as OmniNet. For example, a single instance of OmniNet can concurrently learn to perform the tasks of part-of-speech tagging, image captioning, visual question answering and video activity recognition. We demonstrate that training these four tasks together results in about three times compressed model while retaining the performance in comparison to training them individually. We also show that using this neural network pre-trained on some modalities assists in learning unseen tasks such as video captioning and video question answering. This illustrates the generalization capacity of the self-attention mechanism on the spatio-temporal cache present in OmniNet.

arXiv commentSource code available at: https://github.com/subho406/OmniNet

References in corpus (11)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Two-Stream Convolutional Networks for Action Recognition in Videos
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• A DIRT-T Approach to Unsupervised Domain Adaptation
• One Model To Learn Them All
• Multimodal Residual Learning for Visual QA
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages
• Exploring Neural Transducers for End-to-End Speech Recognition
• Hierarchical Multitask Learning for CTC-based Speech Recognition

Cited by in corpus (7)

• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Multi-Task Learning with Deep Neural Networks: A Survey
• A Recurrent Vision-and-Language BERT for Navigation
• 12-in-1: Multi-Task Vision and Language Representation Learning
• Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning