Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments

Abstract

The capability of a reinforcement learning (RL) agent heavily depends on the diversity of the learning scenarios generated by the environment. Generation of diverse realistic scenarios is challenging for real-time strategy (RTS) environments. The RTS environments are characterized by intelligent entities/non-RL agents cooperating and competing with the RL agents with large state and action spaces over a long period of time, resulting in an infinite space of feasible, but not necessarily realistic, scenarios involving complex interaction among different RL and non-RL agents. Yet, most of the existing simulators rely on randomly generating the environments based on predefined settings/layouts and offer limited flexibility and control over the environment dynamics for researchers to generate diverse, realistic scenarios as per their demand. To address this issue, for the first time, we formally introduce the benefits of adopting an existing formal scenario specification language, SCENIC, to assist researchers to model and generate diverse scenarios in an RTS environment in a flexible, systematic, and programmatic manner. To showcase the benefits, we interfaced SCENIC to an existing RTS environment Google Research Football(GRF) simulator and introduced a benchmark consisting of 32 realistic scenarios, encoded in SCENIC, to train RL agents and testing their generalization capabilities. We also show how researchers/RL practitioners can incorporate their domain knowledge to expedite the training process by intuitively modeling stochastic programmatic policies with SCENIC.

arXiv commentFirst two authors contributed equally. The final version of this paper is accepted at Proceedings of the AAAI Conference on Artificial Intelligence, 36(6), 6028-6036. https://doi.org/10.1609/aaai.v36i6.20549

References in corpus (14)

• Trust Region Policy Optimization
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• DeepMind Control Suite
• Deep Reinforcement Learning that Matters
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• dm_control: Software and Tasks for Continuous Control
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Natural Environment Benchmarks for Reinforcement Learning
• Paracosm: A Language and Tool for Testing Autonomous Driving Systems
• RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning