What Would Jiminy Cricket Do? Towards Agents That Behave Morally
arXiv:2110.13136
Abstract
When making everyday decisions, people are guided by their conscience, an internal sense of right and wrong. By contrast, artificial agents are currently not endowed with a moral sense. As a consequence, they may learn to behave immorally when trained on environments that ignore moral concerns, such as violent video games. With the advent of generally capable agents that pretrain on many environments, it will become necessary to mitigate inherited biases from environments that teach immoral behavior. To facilitate the development of agents that avoid causing wanton harm, we introduce Jiminy Cricket, an environment suite of 25 text-based adventure games with thousands of diverse, morally salient scenarios. By annotating every possible game state, the Jiminy Cricket environments robustly evaluate whether agents can act morally while maximizing reward. Using models with commonsense moral knowledge, we create an elementary artificial conscience that assesses and guides agents. In extensive experiments, we find that the artificial conscience approach can steer agents towards moral behavior without sacrificing performance.
NeurIPS 2021. Environments available here https://github.com/hendrycks/jiminy-cricket
References in corpus (7)
- Decision Transformer: Reinforcement Learning via Sequence Modeling
- AI Safety Gridworlds
- Constrained Policy Optimization
- Aligning AI With Shared Human Values
- Open-Ended Learning Leads to Generally Capable Agents
- Graph Constrained Reinforcement Learning for Natural Language Action Spaces
- How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worlds