Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions

Abstract

Large language models (LLMs) can be used to generate text data for training and evaluating other models. However, creating high-quality datasets with LLMs can be challenging. In this work, we explore human-AI partnerships to facilitate high diversity and accuracy in LLM-based text data generation. We first examine two approaches to diversify text generation: 1) logit suppression, which minimizes the generation of languages that have already been frequently generated, and 2) temperature sampling, which flattens the token sampling probability. We found that diversification approaches can increase data diversity but often at the cost of data accuracy (i.e., text and labels being appropriate for the target domain). To address this issue, we examined two human interventions, 1) label replacement (LR), correcting misaligned labels, and 2) out-of-scope filtering (OOSF), removing instances that are out of the user's domain of interest or to which no considered label applies. With oracle studies, we found that LR increases the absolute accuracy of models trained with diversified datasets by 14.4%. Moreover, we found that some models trained with data generated with LR interventions outperformed LLM-based few-shot classification. In contrast, OOSF was not effective in increasing model accuracy, implying the need for future work in human-in-the-loop text data generation.

arXiv commentAccepted as a long paper at ACL 2023

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• mixup: Beyond Empirical Risk Minimization
• Character-level Convolutional Networks for Text Classification
• Data Augmentation using Pre-trained Transformer Models
• Towards Understanding Knowledge Distillation
• Data Boost: Text Data Augmentation Through Reinforcement Learning Guided Conditional Generation
• CG-BERT: Conditional Text Generation with BERT for Generalized Few-shot Intent Detection
• SQUINKY! A Corpus of Sentence-level Formality, Informativeness, and Implicature
• Red Teaming Language Models with Language Models
• iSEA: An Interactive Pipeline for Semantic Error Analysis of NLP Models

Cited by in corpus (4)

• Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Search
• Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbots
• BioSpark: Beyond Analogical Inspiration to LLM-augmented Transfer
• Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Prevention