Neural Composition: Learning to Generate from Multiple Models
arXiv:2007.16013
Abstract
Decomposing models into multiple components is critically important in many applications such as language modeling (LM) as it enables adapting individual components separately and biasing of some components to the user's personal preferences. Conventionally, contextual and personalized adaptation for language models, are achieved through class-based factorization, which requires class-annotated data, or through biasing to individual phrases which is limited in scale. In this paper, we propose a system that combines model-defined components, by learning when to activate the generation process from each individual component, and how to combine probability distributions from each component, directly from unlabeled text data.
arXiv commentSelf-Supervised Learning for Speech and Audio Processing Workshop @ NeurIPS 2020
References in corpus (7)
- Adam: A Method for Stochastic Optimization
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Sequence Transduction with Recurrent Neural Networks
- Listen, Attend and Spell
- Personalized Speech recognition on mobile devices
- Deep context: end-to-end contextual speech recognition
- End-to-end contextual speech recognition using class language models and a token passing decoder