Russian Natural Language Generation: Creation of a Language Modelling Dataset and Evaluation with Modern Neural Architectures

Abstract

Generating coherent, grammatically correct, and meaningful text is very challenging, however, it is crucial to many modern NLP systems. So far, research has mostly focused on English language, for other languages both standardized datasets, as well as experiments with state-of-the-art models, are rare. In this work, we i) provide a novel reference dataset for Russian language modeling, ii) experiment with popular modern methods for text generation, namely variational autoencoders, and generative adversarial networks, which we trained on the new dataset. We evaluate the generated text regarding metrics such as perplexity, grammatical correctness and lexical diversity.

References in corpus (5)

• Pointer Sentinel Mixture Models
• Regularizing and Optimizing LSTM Language Models
• Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
• GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution
• Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing