Policy Learning Using Weak Supervision
arXiv:2010.01748
Abstract
Most existing policy learning solutions require the learning agents to receive high-quality supervision signals such as well-designed rewards in reinforcement learning (RL) or high-quality expert demonstrations in behavioral cloning (BC). These quality supervisions are usually infeasible or prohibitively expensive to obtain in practice. We aim for a unified framework that leverages the available cheap weak supervisions to perform policy learning efficiently. To handle this problem, we treat the "weak supervision" as imperfect information coming from a peer agent, and evaluate the learning agent's policy based on a "correlated agreement" with the peer agent's policy (instead of simple agreements). Our approach explicitly punishes a policy for overfitting to the weak supervision. In addition to theoretical guarantees, extensive evaluations on tasks including RL with noisy rewards, BC with weak demonstrations, and standard policy co-training show that our method leads to substantial performance improvements, especially when the complexity or the noise of the learning environments is high.
arXiv commentNeurIPS 2021
References in corpus (15)
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- End to End Learning for Self-Driving Cars
- Dueling Network Architectures for Deep Reinforcement Learning
- Classification with Noisy Labels by Importance Reweighting
- Solving Rubik's Cube with a Robot Hand
- Training Convolutional Networks with Noisy Labels
- Deep Q-learning from Demonstrations
- Reinforcement Learning from Imperfect Demonstrations
- DART: Noise Injection for Robust Imitation Learning
- Provably End-to-end Label-Noise Learning without Anchor Points
- Clusterability as an Alternative to Anchor Points When Learning with Noisy Labels
- Learning in the Presence of Corruption
- Understanding Instance-Level Label Noise: Disparate Impacts and Treatments
- Hybrid Reinforcement Learning with Expert State Sequences