A Large-Scale Chinese Short-Text Conversation Dataset

Abstract

The advancements of neural dialogue generation models show promising results on modeling short-text conversations. However, training such models usually needs a large-scale high-quality dialogue corpus, which is hard to access. In this paper, we present a large-scale cleaned Chinese conversation dataset, LCCC, which contains a base version (6.8million dialogues) and a large version (12.0 million dialogues). The quality of our dataset is ensured by a rigorous data cleaning pipeline, which is built based on a set of rules and a classifier that is trained on manually annotated 110K dialogue pairs. We also release pre-training dialogue models which are trained on LCCC-base and LCCC-large respectively. The cleaned dataset and the pre-training models will facilitate the research of short-text conversation modeling. All the models and datasets are available at https://github.com/thu-coai/CDial-GPT.

arXiv commentAccepted by NLPCC 2020 (Best Student Paper)

References in corpus (19)

• Sequence to Sequence Learning with Neural Networks
• Pre-trained Models for Natural Language Processing: A Survey
• The Curious Case of Neural Text Degeneration
• DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
• Wizard of Wikipedia: Knowledge-Powered Conversational agents
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
• Towards a Human-like Open-Domain Chatbot
• The Evolved Transformer
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• A Survey of Available Corpora for Building Data-Driven Dialogue Systems
• A Persona-Based Neural Conversation Model
• DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
• Personalized Dialogue Generation with Diversified Traits
• A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
• Proactive Human-Machine Conversation with Explicit Conversation Goals
• Training Millions of Personalized Dialogue Agents
• An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation
• A Dataset for Document Grounded Conversations