Agent57: Outperforming the Atari Human Benchmark

Abstract

Atari games have been a long-standing benchmark in the reinforcement learning (RL) community for the past decade. This benchmark was proposed to test general competency of RL algorithms. Previous work has achieved good average performance by doing outstandingly well on many games of the set, but very poorly in several of the most challenging games. We propose Agent57, the first deep RL agent that outperforms the standard human benchmark on all 57 Atari games. To achieve this result, we train a neural network which parameterizes a family of policies ranging from very exploratory to purely exploitative. We propose an adaptive mechanism to choose which policy to prioritize throughout the training process. Additionally, we utilize a novel parameterization of the architecture that allows for more consistent and stable learning.

References in corpus (26)

• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Noisy Networks for Exploration
• Distributed Prioritized Experience Replay
• Parameter Space Noise for Exploration
• Count-Based Exploration with Neural Density Models
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Learning Dexterous In-Hand Manipulation
• Exploration by Random Network Distillation
• Go-Explore: a New Approach for Hard-Exploration Problems
• Episodic Curiosity through Reachability
• On Upper-Confidence Bound Policies for Non-Stationary Bandit Problems
• Variational Intrinsic Control
• Meta-Gradient Reinforcement Learning
• RUDDER: Return Decomposition for Delayed Rewards
• Observe and Look Further: Achieving Consistent Performance on Atari
• Never Give Up: Learning Directed Exploration Strategies
• Variational Option Discovery Algorithms
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Contingency-Aware Exploration in Reinforcement Learning
• Generalization of Reinforcement Learners with Working and Episodic Memory
• Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
• Hindsight Credit Assignment
• Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning
• Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
• Sparse Attentive Backtracking: Long-Range Credit Assignment in Recurrent Networks
• Adapting Behaviour for Learning Progress

Cited by in corpus (77)

• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• First return, then explore
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models
• Variational Quantum Reinforcement Learning via Evolutionary Optimization
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Celebrating Diversity in Shared Multi-Agent Reinforcement Learning
• Effective Diversity in Population Based Reinforcement Learning
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Quantum Architecture Search via Deep Reinforcement Learning
• Grounded Language Learning Fast and Slow
• Uncovering Instabilities in Variational-Quantum Deep Q-Networks
• EpidemiOptim: A Toolbox for the Optimization of Control Policies in Epidemiological Models
• Mastering Atari with Discrete World Models
• Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning
• PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goals
• BeBold: Exploration Beyond the Boundary of Explored Regions
• Reinforcement Learning with Quantum Variational Circuits
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
• Tactical Optimism and Pessimism for Deep Reinforcement Learning
• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Learning to run a Power Network Challenge: a Retrospective Analysis
• Synthetic Returns for Long-Term Credit Assignment
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• Behavior From the Void: Unsupervised Active Pre-Training
• Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning
• Fever Basketball: A Complex, Flexible, and Asynchronized Sports Game Environment for Multi-agent Reinforcement Learning
• Self-Imitation Advantage Learning
• Return-based Scaling: Yet Another Normalisation Trick for Deep RL
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• Convergent and Efficient Deep Q Network Algorithm
• Interferobot: aligning an optical interferometer by a reinforcement learning agent
• Modeling Penetration Testing with Reinforcement Learning Using Capture-the-Flag Challenges: Trade-offs between Model-free Learning and A Priori Knowledge
• Domain-Level Explainability -- A Challenge for Creating Trust in Superhuman AI Strategies
• Reward Tweaking: Maximizing the Total Reward While Planning for Short Horizons
• A learning gap between neuroscience and reinforcement learning
• Heatmap-based Object Detection and Tracking with a Fully Convolutional Neural Network
• On Catastrophic Interference in Atari 2600 Games
• CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning
• Towards Automatic Actor-Critic Solutions to Continuous Control
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learning
• The Architectural Implications of Distributed Reinforcement Learning on CPU-GPU Systems
• Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation
• Temporally-Extended ε-Greedy Exploration
• Guided Exploration with Proximal Policy Optimization using a Single Demonstration
• Group Equivariant Deep Reinforcement Learning
• Pre-trained Word Embeddings for Goal-conditional Transfer Learning in Reinforcement Learning
• Distributed Deep Reinforcement Learning: An Overview
• Reinforcement Learning with Latent Flow
• Reasoning-Modulated Representations
• There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning
• Interactive Machine Comprehension with Dynamic Knowledge Graphs
• An Entropy Regularization Free Mechanism for Policy-based Reinforcement Learning
• ADER:Adapting between Exploration and Robustness for Actor-Critic Methods
• Instance based Generalization in Reinforcement Learning
• Evolutionary Self-Replication as a Mechanism for Producing Artificial Intelligence
• Benchmarking the Spectrum of Agent Capabilities
• Transfer Learning and Curriculum Learning in Sokoban
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Detecting Rewards Deterioration in Episodic Reinforcement Learning
• A Novel Approach to Curiosity and Explainable Reinforcement Learning via Interpretable Sub-Goals
• A brain basis of dynamical intelligence for AI and computational neuroscience
• Model-free Policy Learning with Reward Gradients
• Did I do that? Blame as a means to identify controlled effects in reinforcement learning
• Robot in a China Shop: Using Reinforcement Learning for Location-Specific Navigation Behaviour
• When should agents explore?
• Contrastive Active Inference
• A Proposal for Intelligent Agents with Episodic Memory
• Inducing Functions through Reinforcement Learning without Task Specification
• Human-Level Control without Server-Grade Hardware
• Fixed $\beta$-VAE Encoding for Curious Exploration in Complex 3D Environments
• High Performance Across Two Atari Paddle Games Using the Same Perceptual Control Architecture Without Training
• Planning and Learning Using Adaptive Entropy Tree Search