Intrinsic fluctuations of reinforcement learning promote cooperation
arXiv:2209.01013 · doi:10.1038/s41598-023-27672-7
Abstract
In this work, we ask for and answer what makes classical temporal-difference reinforcement learning with epsilon-greedy strategies cooperative. Cooperating in social dilemma situations is vital for animals, humans, and machines. While evolutionary theory revealed a range of mechanisms promoting cooperation, the conditions under which agents learn to cooperate are contested. Here, we demonstrate which and how individual elements of the multi-agent learning setting lead to cooperation. We use the iterated Prisoner's dilemma with one-period memory as a testbed. Each of the two learning agents learns a strategy that conditions the following action choices on both agents' action choices of the last round. We find that next to a high caring for future rewards, a low exploration rate, and a small learning rate, it is primarily intrinsic stochastic fluctuations of the reinforcement learning process which double the final rate of cooperation to up to 80%. Thus, inherent noise is not a necessary evil of the iterative learning process. It is a critical asset for the learning of cooperation. However, we also point out the trade-off between a high likelihood of cooperative behavior and achieving this in a reasonable amount of time. Our findings are relevant for purposefully designing cooperative algorithms and regulating undesired collusive effects.
arXiv comment21 pages, 7 figures
References in corpus (14)
- Statistical physics of human cooperation
- Evolutionary dynamics of group interactions on structured populations: A review
- Cooperation in noisy case: prisoner's dilemma game on two types of regular random graphs
- Selection of noise level in strategy adoption for spatial social dilemmas
- Cooperation dilemma in finite populations under fluctuating environments
- Inequity aversion improves cooperation in intertemporal social dilemmas
- Deterministic limit of temporal difference reinforcement learning for stochastic games
- Intrinsic noise in game dynamical learning
- Numerical analysis of a reinforcement learning model with the dynamic aspiration level in the iterated Prisoner's Dilemma
- Learning dynamics explains human behavior in Prisoner's Dilemma on networks
- Symmetric equilibrium of multi-agent reinforcement learning in repeated prisoner's dilemma
- Modeling the effects of environmental and perceptual uncertainty using deterministic reinforcement learning dynamics with partial observability
- The physics of governance networks: critical transitions in contagion dynamics on multilayer adaptive networks with application to the sustainable use of renewable resources
- Learning to play public good games