Code-switched inspired losses for generic spoken dialog representations
arXiv:2108.12465
Abstract
Spoken dialog systems need to be able to handle both multiple languages and multilinguality inside a conversation (\textit{e.g} in case of code-switching). In this work, we introduce new pretraining losses tailored to learn multilingual spoken dialog representations. The goal of these losses is to expose the model to code-switched language. To scale up training, we automatically build a pretraining corpus composed of multilingual conversations in five different languages (French, Italian, English, German and Spanish) from \texttt{OpenSubtitles}, a huge multilingual corpus composed of 24.3G tokens. We test the generic representations on \texttt{MIAM}, a new benchmark composed of five dialog act corpora on the same aforementioned languages as well as on two novel multilingual downstream tasks (\textit{i.e} multilingual mask utterance retrieval and multilingual inconsistency identification). Our experiments show that our new code switched-inspired losses achieve a better performance in both monolingual and multilingual settings.
References in corpus (23)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Gaussian Error Linear Units (GELUs)
- BERTScore: Evaluating Text Generation with BERT
- Cross-lingual Language Model Pretraining
- A Structured Self-attentive Sentence Embedding
- Exploiting Similarities among Languages for Machine Translation
- Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech
- DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
- Multilingual Denoising Pre-training for Neural Machine Translation
- mT5: A massively multilingual pre-trained text-to-text transformer
- Multi-Task Learning as Multi-Objective Optimization
- BilBOWA: Fast Bilingual Distributed Representations without Word Alignments
- Evaluation of Text Generation: A Survey
- Cross-Lingual Ability of Multilingual BERT: An Empirical Study
- DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
- Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation
- HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization
- From the Token to the Review: A Hierarchical Multimodal approach to Opinion Mining
- Training Multilingual Machine Translation by Alternately Freezing Language-Specific Encoders-Decoders
- Affect-Driven Dialog Generation
- Guiding attention in Sequence-to-sequence models for Dialogue Act prediction
- ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation