A General Language Assistant as a Laboratory for Alignment

Abstract

Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. As an initial foray in this direction we study simple baseline techniques and evaluations, such as prompting. We find that the benefits from modest interventions increase with model size, generalize to a variety of alignment evaluations, and do not compromise the performance of large models. Next we investigate scaling trends for several training objectives relevant to alignment, comparing imitation learning, binary discrimination, and ranked preference modeling. We find that ranked preference modeling performs much better than imitation learning, and often scales more favorably with model size. In contrast, binary discrimination typically performs and scales very similarly to imitation learning. Finally we study a `preference model pre-training' stage of training, with the goal of improving sample efficiency when finetuning on human preferences.

arXiv comment26+19 pages; v2 typos fixed, refs added, figure scale / colors fixed; v3 correct very non-standard TruthfulQA formatting and metric, alignment implications slightly improved

References in corpus (20)

• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Deep reinforcement learning from human preferences
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Generative Adversarial Imitation Learning
• Unsolved Problems in ML Safety
• Generating Wikipedia by Summarizing Long Sequences
• Finetuned Language Models Are Zero-Shot Learners
• A Constructive Prediction of the Generalization Error Across Scales
• Imitating Interactive Intelligence
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Learning to summarize from human feedback
• Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
• AI safety via debate
• Program Synthesis with Large Language Models
• Supervising strong learners by amplifying weak experts
• Training Verifiers to Solve Math Word Problems
• Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
• Scaling Scaling Laws with Board Games
• ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning

Cited by in corpus (4)

• A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias
• Ethics of generative AI and manipulation: a design-oriented research agenda
• Can AI Rely on the Systematicity of Truth? The Challenge of Modelling Normative Domains
• On the attribution of confidence to large language models