Transfer Learning for Speech and Language Processing
arXiv:1511.06066
Abstract
Transfer learning is a vital technique that generalizes models trained for one setting or task to other settings or tasks. For example in speech recognition, an acoustic model trained for one language can be used to recognize speech in another language, with little or no re-training data. Transfer learning is closely related to multi-task learning (cross-lingual vs. multilingual), and is traditionally studied in the name of `model adaptation'. Recent advance in deep learning shows that transfer learning becomes much easier and more effective with high-level abstract features learned by deep models, and the `transfer' can be conducted not only between data distributions and data types, but also between model structures (e.g., shallow nets and deep nets) or even model types (e.g., Bayesian models and neural models). This review paper summarizes some recent prominent research towards this direction, particularly for speech and language processing. We also report some results from our group and highlight the potential of this very interesting research field.
arXiv comment13 pages, APSIPA 2015
References in corpus (15)
- Efficient Estimation of Word Representations in Vector Space
- Distilling the Knowledge in a Neural Network
- Natural Language Processing (almost) from Scratch
- Learning Transferable Features with Deep Adaptation Networks
- FitNets: Hints for Thin Deep Nets
- Do Deep Nets Really Need to be Deep?
- Exploiting Similarities among Languages for Machine Translation
- Theoretical Models of Learning to Learn
- Zero-Shot Learning Through Cross-Modal Transfer
- Recurrent Neural Network Training with Dark Knowledge Transfer
- Multilingual Models for Compositional Distributed Semantics
- Transferring Knowledge from a RNN to a DNN
- Unsupervised Learning on Neural Network Outputs: with Application in Zero-shot Learning
- Distilling Word Embeddings: An Encoding Approach
- Knowledge Transfer Pre-training
Cited by in corpus (14)
- How Transferable are Neural Networks in NLP Applications?
- Transfer Learning for Named-Entity Recognition with Neural Networks
- Neuromorphic Hardware learns to learn
- Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0
- Towards Language-Universal End-to-End Speech Recognition
- Discriminative Joint Probability Maximum Mean Discrepancy (DJP-MMD) for Domain Adaptation
- Cantonese Automatic Speech Recognition Using Transfer Learning from Mandarin
- Two-Staged Acoustic Modeling Adaption for Robust Speech Recognition by the Example of German Oral History Interviews
- Fine-Grained Entity Type Classification by Jointly Learning Representations and Label Embeddings
- Modelling Domain Relationships for Transfer Learning on Retrieval-based Question Answering Systems in E-commerce
- Phonetic Temporal Neural Model for Language Identification
- Deep Factorization for Speech Signal
- AP19-OLR Challenge: Three Tasks and Their Baselines
- Data Generation Using Pass-phrase-dependent Deep Auto-encoders for Text-Dependent Speaker Verification