Unified Models of Human Behavioral Agents in Bandits, Contextual Bandits and RL

Abstract

Artificial behavioral agents are often evaluated based on their consistent behaviors and performance to take sequential actions in an environment to maximize some notion of cumulative reward. However, human decision making in real life usually involves different strategies and behavioral trajectories that lead to the same empirical outcome. Motivated by clinical literature of a wide range of neurological and psychiatric disorders, we propose here a more general and flexible parametric framework for sequential decision making that involves a two-stream reward processing mechanism. We demonstrated that this framework is flexible and unified enough to incorporate a family of problems spanning multi-armed bandits (MAB), contextual bandits (CB) and reinforcement learning (RL), which decompose the sequential decision making process in different levels. Inspired by the known reward processing abnormalities of many mental disorders, our clinically-inspired agents demonstrated interesting behavioral trajectories and comparable performance on simulated tasks with particular reward distributions, a real-world dataset capturing human decision-making in gambling tasks, and the PacMan game across different reward stationarities in a lifelong learning setting.

arXiv commentProceeding of HBAI 2020. This article supersedes and extends our work arXiv:1706.02897 (MAB) and arXiv:1906.11286 (RL) into the Contextual Bandit (CB) framework. It generalized extensively into multi-armed bandits, contextual bandits and RL settings to create a unified framework of human behavioral agents

References in corpus (5)

• Analysis of Thompson Sampling for the multi-armed bandit problem
• Thompson Sampling for Contextual Bandits with Linear Payoffs
• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
• Contextual Bandit Algorithms with Supervised Learning Guarantees
• Bandit Models of Human Behavior: Reward Processing in Mental Disorders

Cited by in corpus (10)

• Predicting human decision making in psychological tasks with recurrent neural networks
• A Story of Two Streams: Reinforcement Learning Models from Human Behavior and Neuropsychiatry
• Speaker Diarization as a Fully Online Learning Problem in MiniVox
• Online learning with Corrupted context: Corrupted Contextual Bandits
• Online Semi-Supervised Learning with Bandit Feedback
• Contextual Bandit with Missing Rewards
• Spectral Clustering using Eigenspectrum Shape Based Nystrom Sampling
• Computing the Dirichlet-Multinomial Log-Likelihood Function
• Online Semi-Supervised Learning in Contextual Bandits with Episodic Reward
• Etat de l'art sur l'application des bandits multi-bras