EasyTransfer -- A Simple and Scalable Deep Transfer Learning Platform for NLP Applications
arXiv:2011.09463
Abstract
The literature has witnessed the success of leveraging Pre-trained Language Models (PLMs) and Transfer Learning (TL) algorithms to a wide range of Natural Language Processing (NLP) applications, yet it is not easy to build an easy-to-use and scalable TL toolkit for this purpose. To bridge this gap, the EasyTransfer platform is designed to develop deep TL algorithms for NLP applications. EasyTransfer is backended with a high-performance and scalable engine for efficient training and inference, and also integrates comprehensive deep TL algorithms, to make the development of industrial-scale TL applications easier. In EasyTransfer, the built-in data and model parallelism strategies, combined with AI compiler optimization, show to be 4.0x faster than the community version of distributed training. EasyTransfer supports various NLP models in the ModelZoo, including mainstream PLMs and multi-modality models. It also features various in-house developed TL algorithms, together with the AppZoo for NLP applications. The toolkit is convenient for users to quickly start model training, evaluation, and online deployment. EasyTransfer is currently deployed at Alibaba to support a variety of business scenarios, including item recommendation, personalized search, conversational question answering, etc. Extensive experiments on real-world datasets and online applications show that EasyTransfer is suitable for online production with cutting-edge performance for various applications. The source code of EasyTransfer is released at Github (https://github.com/alibaba/EasyTransfer).
arXiv comment11 pages
References in corpus (17)
- Distilling the Knowledge in a Neural Network
- Longformer: The Long-Document Transformer
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- Linformer: Self-Attention with Linear Complexity
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- Generating Long Sequences with Sparse Transformers
- Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
- Wasserstein Distance Guided Representation Learning for Domain Adaptation
- A Comprehensive Survey on Transfer Learning
- Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks
- Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
- How Transferable are Neural Networks in NLP Applications?
- A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
- FusionStitching: Boosting Memory Intensive Computations for Deep Learning Workloads
- FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrieval
- Meta Fine-Tuning Neural Language Models for Multi-Domain Text Mining