Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control

Abstract

Pretrained language models have demonstrated extraordinary capabilities in language generation. However, real-world tasks often require controlling the distribution of generated text in order to mitigate bias, promote fairness, and achieve personalization. Existing techniques for controlling the distribution of generated text only work with quantified distributions, which require pre-defined categories, proportions of the distribution, or an existing corpus following the desired distributions. However, many important distributions, such as personal preferences, are unquantified. In this work, we tackle the problem of generating text following arbitrary distributions (quantified and unquantified) by proposing Nano, a few-shot human-in-the-loop training algorithm that continuously learns from human feedback. Nano achieves state-of-the-art results on single topic/attribute as well as quantified distribution control compared to previous works. We also show that Nano is able to learn unquantified distributions, achieves personalization, and captures differences between different individuals' personal preferences with high sample efficiency.

Accepted to ACL Findings 2023

References in corpus (14)

• Continuous control with deep reinforcement learning
• Training language models to follow instructions with human feedback
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• A Survey of Human-in-the-loop for Machine Learning
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Fine-Tuning Language Models from Human Preferences
• Neural Text Generation with Unlikelihood Training
• Towards Understanding and Mitigating Social Biases in Language Models
• Quark: Controllable Text Generation with Reinforced Unlearning
• A Distributional Approach to Controlled Text Generation
• When and Why does a Model Fail? A Human-in-the-loop Error Detection Framework for Sentiment Analysis
• Reinforced Generative Adversarial Network for Abstractive Text Summarization
• Personalized sentence generation using generative adversarial networks with author-specific word usage
• GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation