Meta-Learning for Contextual Bandit Exploration

Abstract

We describe MELEE, a meta-learning algorithm for learning a good exploration policy in the interactive contextual bandit setting. Here, an algorithm must take actions based on contexts, and learn based only on a reward signal from the action taken, thereby generating an exploration/exploitation trade-off. MELEE addresses this trade-off by learning a good exploration strategy for offline tasks based on synthetic data, on which it can simulate the contextual bandit setting. Based on these simulations, MELEE uses an imitation learning strategy to learn a good exploration policy that can then be applied to true contextual bandit tasks at test time. We compare MELEE to seven strong baseline contextual bandit algorithms on a set of three hundred real-world datasets, on which it outperforms alternatives in most settings, especially when differences in rewards are large. Finally, we demonstrate the importance of having a rich feature representation for learning how to explore.

References in corpus (12)

• Scikit-learn: Machine Learning in Python
• Neural Architecture Search with Reinforcement Learning
• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Learning to Optimize
• Learning Active Learning from Data
• Reinforcement and Imitation Learning via Interactive No-Regret Learning
• Learning to Search Better Than Your Teacher
• Active One-shot Learning
• Learning Algorithms for Active Learning
• Learning to Explore with Meta-Policy Gradient

Cited by in corpus (8)

• Meta Dynamic Pricing: Transfer Learning Across Experiments
• Hyper-parameter Tuning for the Contextual Bandit
• Syndicated Bandits: A Framework for Auto Tuning Hyper-parameters in Contextual Bandit Algorithms
• MAME : Model-Agnostic Meta-Exploration
• AutoML for Contextual Bandits
• Learning to Actively Learn: A Robust Approach
• DORB: Dynamically Optimizing Multiple Rewards with Bandits
• Learning Efficient and Effective Exploration Policies with Counterfactual Meta Policy