General Characterization of Agents by States they Visit

Abstract

Behavioural characterizations (BCs) of decision-making agents, or their policies, are used to study outcomes of training algorithms and as part of the algorithms themselves to encourage unique policies, match expert policy or restrict changes to policy per update. However, previously presented solutions are not applicable in general, either due to lack of expressive power, computational constraint or constraints on the policy or environment. Furthermore, many BCs rely on the actions of policies. We discuss and demonstrate how these BCs can be misleading, especially in stochastic environments, and propose a novel solution based on what states policies visit. We run experiments to evaluate the quality of the proposed BC against baselines and evaluate their use in studying training algorithms, novelty search and trust-region policy optimization. The code is available at https://github.com/miffyli/policy-supervectors.

arXiv commentDeep Reinforcement Learning Workshop, NeurIPS 2021

References in corpus (13)

• Scikit-learn: Machine Learning in Python
• Auto-Encoding Variational Bayes
• Trust Region Policy Optimization
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Deep Reinforcement Learning that Matters
• Go-Explore: a New Approach for Hard-Exploration Problems
• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• Learning Policy Representations in Multiagent Systems
• Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning
• Evaluating the Performance of Reinforcement Learning Algorithms
• Policy Evaluation Networks
• Evaluating Agents without Rewards