Transfer Learning for Speech and Language Processing

Abstract

Transfer learning is a vital technique that generalizes models trained for one setting or task to other settings or tasks. For example in speech recognition, an acoustic model trained for one language can be used to recognize speech in another language, with little or no re-training data. Transfer learning is closely related to multi-task learning (cross-lingual vs. multilingual), and is traditionally studied in the name of `model adaptation'. Recent advance in deep learning shows that transfer learning becomes much easier and more effective with high-level abstract features learned by deep models, and the `transfer' can be conducted not only between data distributions and data types, but also between model structures (e.g., shallow nets and deep nets) or even model types (e.g., Bayesian models and neural models). This review paper summarizes some recent prominent research towards this direction, particularly for speech and language processing. We also report some results from our group and highlight the potential of this very interesting research field.

13 pages, APSIPA 2015

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Natural Language Processing (almost) from Scratch
• Learning Transferable Features with Deep Adaptation Networks
• FitNets: Hints for Thin Deep Nets
• Theoretical Models of Learning to Learn
• Zero-Shot Learning Through Cross-Modal Transfer
• Multilingual Models for Compositional Distributed Semantics
• Transferring Knowledge from a RNN to a DNN
• Knowledge Transfer Pre-training

Cited by in corpus (14)

• How Transferable are Neural Networks in NLP Applications?
• Transfer Learning for Named-Entity Recognition with Neural Networks
• Neuromorphic Hardware learns to learn
• Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0
• Towards Language-Universal End-to-End Speech Recognition
• Discriminative Joint Probability Maximum Mean Discrepancy (DJP-MMD) for Domain Adaptation
• Cantonese Automatic Speech Recognition Using Transfer Learning from Mandarin
• Two-Staged Acoustic Modeling Adaption for Robust Speech Recognition by the Example of German Oral History Interviews
• Fine-Grained Entity Type Classification by Jointly Learning Representations and Label Embeddings
• Modelling Domain Relationships for Transfer Learning on Retrieval-based Question Answering Systems in E-commerce
• Phonetic Temporal Neural Model for Language Identification
• Deep Factorization for Speech Signal
• AP19-OLR Challenge: Three Tasks and Their Baselines
• Data Generation Using Pass-phrase-dependent Deep Auto-encoders for Text-Dependent Speaker Verification