Evaluating the Robustness of Collaborative Agents

Abstract

In order for agents trained by deep reinforcement learning to work alongside humans in realistic settings, we will need to ensure that the agents are \emph{robust}. Since the real world is very diverse, and human behavior often changes in response to agent deployment, the agent will likely encounter novel situations that have never been seen during training. This results in an evaluation challenge: if we cannot rely on the average training or validation reward as a metric, then how can we effectively evaluate robustness? We take inspiration from the practice of \emph{unit testing} in software engineering. Specifically, we suggest that when designing AI agents that collaborate with humans, designers should search for potential edge cases in \emph{possible partner behavior} and \emph{possible states encountered}, and write tests which check that the behavior of the agent in these edge cases is reasonable. We apply this methodology to build a suite of unit tests for the Overcooked-AI environment, and use this test suite to evaluate three proposals for improving robustness. We find that the test suite provides significant insight into the effects of these proposals that were generally not revealed by looking solely at the average validation reward.

References in corpus (17)

• Dota 2 with Large Scale Deep Reinforcement Learning
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Solving Rubik's Cube with a Robot Hand
• Robust Adversarial Reinforcement Learning
• Towards a Human-like Open-Domain Chatbot
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models
• On the Utility of Learning about Humans for Human-AI Coordination
• Adversarial Policies: Attacking Deep Reinforcement Learning
• Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures
• Preparing for the Unknown: Learning a Universal Policy with Online System Identification
• Learning to summarize from human feedback
• "Other-Play" for Zero-Shot Coordination
• On the Utility of Model Learning in HRI
• Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games
• Adversarially Guided Self-Play for Adopting Social Conventions
• Estimating the Brittleness of AI: Safety Integrity Levels and the Need for Testing Out-Of-Distribution Performance