Code-switched inspired losses for generic spoken dialog representations
arXiv:2108.12465
Abstract
Spoken dialog systems need to be able to handle both multiple languages and multilinguality inside a conversation (\textit{e.g} in case of code-switching). In this work, we introduce new pretraining losses tailored to learn multilingual spoken dialog representations. The goal of these losses is to expose the model to code-switched language. To scale up training, we automatically build a pretraining corpus composed of multilingual conversations in five different languages (French, Italian, English, German and Spanish) from \texttt{OpenSubtitles}, a huge multilingual corpus composed of 24.3G tokens. We test the generic representations on \texttt{MIAM}, a new benchmark composed of five dialog act corpora on the same aforementioned languages as well as on two novel multilingual downstream tasks (\textit{i.e} multilingual mask utterance retrieval and multilingual inconsistency identification). Our experiments show that our new code switched-inspired losses achieve a better performance in both monolingual and multilingual settings.
References in corpus (11)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Cross-lingual Language Model Pretraining
- A Structured Self-attentive Sentence Embedding
- DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
- Multilingual Denoising Pre-training for Neural Machine Translation
- Cross-Lingual Ability of Multilingual BERT: An Empirical Study
- HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization
- Training Multilingual Machine Translation by Alternately Freezing Language-Specific Encoders-Decoders
- Affect-Driven Dialog Generation
- Guiding attention in Sequence-to-sequence models for Dialogue Act prediction