Multichannel Generative Language Model: Learning All Possible Factorizations Within and Across Channels

Abstract

A channel corresponds to a viewpoint or transformation of an underlying meaning. A pair of parallel sentences in English and French express the same underlying meaning, but through two separate channels corresponding to their languages. In this work, we present the Multichannel Generative Language Model (MGLM). MGLM is a generative joint distribution model over channels. MGLM marginalizes over all possible factorizations within and across all channels. MGLM endows flexible inference, including unconditional generation, conditional generation (where 1 channel is observed and other channels are generated), and partially observed generation (where incomplete observations are spread across all the channels). We experiment with the Multi30K dataset containing English, French, Czech, and German. We demonstrate experiments with unconditional, conditional, and partially conditional generation. We provide qualitative samples sampled unconditionally from the generative joint distribution. We also quantitatively analyze the quality-diversity trade-offs and find MGLM outperforms traditional bilingual discriminative models.

arXiv comment10 pages (+3 appendix), 11 figures, 5 tables. Accepted to Findings of EMNLP 2020

References in corpus (9)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• KERMIT: Generative Insertion-Based Modeling for Sequences
• Non-Monotonic Sequential Text Generation
• Unifying Human and Statistical Evaluation for Natural Language Generation
• Insertion-based Decoding with automatically Inferred Generation Order
• Unsupervised Cross-lingual Word Embedding by Multilingual Neural Language Models
• Jointly Measuring Diversity and Quality in Text Generation Models