Re-evaluating Evaluation

Abstract

Progress in machine learning is measured by careful evaluation on problems of outstanding common interest. However, the proliferation of benchmark suites and environments, adversarial attacks, and other complications has diluted the basic evaluation model by overwhelming researchers with choices. Deliberate or accidental cherry picking is increasingly likely, and designing well-balanced evaluation suites requires increasing effort. In this paper we take a step back and propose Nash averaging. The approach builds on a detailed analysis of the algebraic structure of evaluation in two basic scenarios: agent-vs-agent and agent-vs-task. The key strength of Nash averaging is that it automatically adapts to redundancies in evaluation data, so that results are not biased by the incorporation of easy tasks or weak agents. Nash averaging thus encourages maximally inclusive evaluation -- since there is no harm (computational cost aside) from including all available tasks and agents.

arXiv commentNIPS 2018, final version

References in corpus (10)

• Intriguing properties of neural networks
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Cyclic dominance in evolutionary games: A review
• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• Count-Based Exploration with Neural Density Models
• The Mechanics of n-Player Differentiable Games
• Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
• Learning values across many orders of magnitude
• Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agents
• Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network

Cited by in corpus (13)

• Game-Theoretic Multiagent Reinforcement Learning
• OpenSpiel: A Framework for Reinforcement Learning in Games
• Analysing Results from AI Benchmarks: Key Indicators and How to Obtain Them
• Learning to Play No-Press Diplomacy with Best Response Policy Iteration
• Continual Match Based Training in Pommerman: Technical Report
• Real World Games Look Like Spinning Tops
• Options as responses: Grounding behavioural hierarchies in multi-agent RL
• Scaling Scaling Laws with Board Games
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• The Benchmark Lottery
• Approximate exploitability: Learning a best response in large games
• Accelerated MM Algorithms for Ranking Scores Inference from Comparison Data
• Fractals2019: Combinatorial Optimisation with Dynamic Constraint Annealing