Supervising strong learners by amplifying weak experts
arXiv:1810.08575
Abstract
Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated for a human to directly evaluate. We propose Iterated Amplification, an alternative training strategy which progressively builds up a training signal for difficult problems by combining solutions to easier subproblems. Iterated Amplification is closely related to Expert Iteration (Anthony et al., 2017; Silver et al., 2017), except that it uses no external reward function. We present results in algorithmic environments, showing that Iterated Amplification can efficiently learn complex behaviors.
References in corpus (8)
- Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
- Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
- Deep reinforcement learning from human preferences
- Neural GPUs Learn Algorithms
- Thinking Fast and Slow with Deep Learning and Tree Search
- Making Neural Programming Architectures Generalize via Recursion
- Neural Programmer: Inducing Latent Programs with Gradient Descent
- AI safety via debate
Cited by in corpus (15)
- Fine-Tuning Language Models from Human Preferences
- Scalable agent alignment via reward modeling: a research direction
- Recursively Summarizing Books with Human Feedback
- Alignment of Language Agents
- Learning to summarize from human feedback
- A General Language Assistant as a Laboratory for Alignment
- Truthful AI: Developing and governing AI that does not lie
- Learning Human Objectives by Evaluating Hypothetical Behavior
- Modeling AGI Safety Frameworks with Causal Influence Diagrams
- Evolutionary Computation and AI Safety: Research Problems Impeding Routine and Safe Real-world Application of Evolution
- Avoiding Tampering Incentives in Deep RL via Decoupled Approval
- The Challenge of Value Alignment: from Fairer Algorithms to AI Safety
- A Roadmap for Robust End-to-End Alignment
- AI Alignment vs. AI Ethical Treatment: 10 Challenges
- (When) Is Truth-telling Favored in AI Debate?