One Model To Learn Them All

Abstract

Deep learning yields great results across many fields, from speech recognition, image classification, to translation. But for each problem, getting a deep model to work well involves research into the architecture and a long period of tuning. We present a single model that yields good results on a number of problems spanning multiple domains. In particular, this single model is trained concurrently on ImageNet, multiple translation tasks, image captioning (COCO dataset), a speech recognition corpus, and an English parsing task. Our model architecture incorporates building blocks from multiple domains. It contains convolutional layers, an attention mechanism, and sparsely-gated layers. Each of these computational blocks is crucial for a subset of the tasks we train on. Interestingly, even if a block is not crucial for a task, we observe that adding it never hurts performance and in most cases improves it on all tasks. We also show that tasks with less data benefit largely from joint training with other tasks, while performance on large tasks degrades only slightly if at all.

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Xception: Deep Learning with Depthwise Separable Convolutions
• Neural Machine Translation in Linear Time
• Depthwise Separable Convolutions for Neural Machine Translation
• Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
• Encoding Source Language with Convolutional Neural Network for Machine Translation

Cited by in corpus (77)

• On the Opportunities and Risks of Foundation Models
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches
• Multi-Task Learning with Deep Neural Networks: A Survey
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Software Engineering Challenges of Deep Learning
• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• Self-Supervised MultiModal Versatile Networks
• Self-Supervised Learning for Videos: A Survey
• Pretrained Transformers as Universal Computation Engines
• Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inference
• Global-Locally Self-Attentive Dialogue State Tracker
• Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
• Measuring the Intrinsic Dimension of Objective Landscapes
• Variational Transformers for Diverse Response Generation
• Deep Learning Based Chatbot Models
• Pseudo-task Augmentation: From Deep Multitask Learning to Intratask Sharing---and Back
• Image-Conditioned Graph Generation for Road Network Extraction
• Attention over Parameters for Dialogue Systems
• Towards Universal Object Detection by Domain Attention
• Collaborative Intelligence: Challenges and Opportunities
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension
• Universal Semi-Supervised Semantic Segmentation
• OmniNet: A unified architecture for multi-modal multi-task learning
• Deep Learning for identifying radiogenomic associations in breast cancer
• Multitask Learning Strengthens Adversarial Robustness
• Feature Partitioning for Efficient Multi-Task Architectures
• Navigating the Trade-Off between Multi-Task Learning and Learning to Multitask in Deep Neural Networks
• Machine Translation: A Literature Review
• Dynamic Multi-Level Multi-Task Learning for Sentence Simplification
• Bidirectional Scene Text Recognition with a Single Decoder
• Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approach
• TIMELY: Pushing Data Movements and Interfaces in PIM Accelerators Towards Local and in Time Domain
• Evolutionary Architecture Search For Deep Multitask Networks
• LARNN: Linear Attention Recurrent Neural Network
• MoEL: Mixture of Empathetic Listeners
• Scene Graph Parsing as Dependency Parsing
• Towards General Purpose Vision Systems
• SpeechNet: A Universal Modularized Model for Speech Processing Tasks
• Dynamic-structured Semantic Propagation Network
• Neural Skill Transfer from Supervised Language Tasks to Reading Comprehension
• A Multi-Task Deep Learning Framework for Building Footprint Segmentation
• HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections
• One Deep Music Representation to Rule Them All? : A comparative analysis of different representation learning strategies
• BitTensor: A Peer-to-Peer Intelligence Market
• Computation on Sparse Neural Networks: an Inspiration for Future Hardware
• MultiNet: Multi-Modal Multi-Task Learning for Autonomous Driving
• AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning
• Exploring Uncertainty in Conditional Multi-Modal Retrieval Systems
• Comprehensive and Efficient Data Labeling via Adaptive Model Scheduling
• Enhancing a Neurocognitive Shared Visuomotor Model for Object Identification, Localization, and Grasping With Learning From Auxiliary Tasks
• Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains
• Contextual Classification Using Self-Supervised Auxiliary Models for Deep Neural Networks
• Improved Variational Neural Machine Translation by Promoting Mutual Information
• PolyViT: Co-training Vision Transformers on Images, Videos and Audio
• The Traveling Observer Model: Multi-task Learning Through Spatial Variable Embeddings
• Cross-lingual Data Transformation and Combination for Text Classification
• Multi-Task Deep Learning for Legal Document Translation, Summarization and Multi-Label Classification
• Text is Text, No Matter What: Unifying Text Recognition using Knowledge Distillation
• Multilingual Dialogue Generation with Shared-Private Memory
• Multimodal speech synthesis architecture for unsupervised speaker adaptation
• Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems
• Cortex Neural Network: learning with Neural Network groups
• A Dataset and Benchmarks for Multimedia Social Analysis
• Towards Robust Pattern Recognition: A Review
• Formal Fields: A Framework to Automate Code Generation Across Domains
• CARLS: Cross-platform Asynchronous Representation Learning System
• MULTIMODAL ANALYSIS: Informed content estimation and audio source separation
• Recurrent Stacking of Layers in Neural Networks: An Application to Neural Machine Translation
• Interaction Networks: Using a Reinforcement Learner to train other Machine Learning algorithms
• Deep Unified Multimodal Embeddings for Understanding both Content and Users in Social Media Networks
• Deep Artificial Intelligence for Fantasy Football Language Understanding
• Adaptive Precision Training for Resource Constrained Devices
• One Network Fits All? Modular versus Monolithic Task Formulations in Neural Networks