The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding

Abstract

We present MT-DNN, an open-source natural language understanding (NLU) toolkit that makes it easy for researchers and developers to train customized deep learning models. Built upon PyTorch and Transformers, MT-DNN is designed to facilitate rapid customization for a broad spectrum of NLU tasks, using a variety of objectives (classification, regression, structured prediction) and text encoders (e.g., RNNs, BERT, RoBERTa, UniLM). A unique feature of MT-DNN is its built-in support for robust and transferable learning using the adversarial multi-task learning paradigm. To enable efficient production deployment, MT-DNN supports multi-task knowledge distillation, which can substantially compress a deep neural model without significant performance drop. We demonstrate the effectiveness of MT-DNN on a wide range of NLU applications across general and biomedical domains. The software and pre-trained models will be publicly available at https://github.com/namisan/mt-dnn.

arXiv comment9 pages, 3 figures and 3 tables

References in corpus (15)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Natural Language Processing (almost) from Scratch
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• An Overview of Multi-Task Learning in Deep Neural Networks
• Do Deep Nets Really Need to be Deep?
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Bayesian Dark Knowledge
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• Adversarial Training for Large Neural Language Models
• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
• Adversarial NLI: A New Benchmark for Natural Language Understanding
• Stochastic Answer Networks for Natural Language Inference

Cited by in corpus (7)

• Revisiting Few-sample BERT Fine-tuning
• Adversarial Training for Large Neural Language Models
• Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding
• Adversarial Training for Commonsense Inference
• OCHADAI-KYOTO at SemEval-2021 Task 1: Enhancing Model Generalization and Robustness for Lexical Complexity Prediction
• Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
• Multi-hop Graph Convolutional Network with High-order Chebyshev Approximation for Text Reasoning