Neural Contextual Bandits with Deep Representation and Shallow Exploration

Abstract

We study a general class of contextual bandits, where each context-action pair is associated with a raw feature vector, but the reward generating function is unknown. We propose a novel learning algorithm that transforms the raw feature vector using the last hidden layer of a deep ReLU neural network (deep representation learning), and uses an upper confidence bound (UCB) approach to explore in the last linear layer (shallow exploration). We prove that under standard assumptions, our proposed algorithm achieves $\tilde{O}(\sqrt{T})$ finite-time regret, where $T$ is the learning time horizon. Compared with existing neural contextual bandit algorithms, our approach is computationally much more efficient since it only needs to explore in the last layer of the deep neural network.

arXiv comment28 pages, 1 figure, 1 table

References in corpus (12)

• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Scalable Bayesian Optimization Using Deep Neural Networks
• Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Provably Optimal Algorithms for Generalized Linear Contextual Bandits
• Finite-Time Analysis of Kernelised Contextual Bandits
• Neural Contextual Bandits with UCB-based Exploration
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• Neural Thompson Sampling
• Deep Neural Linear Bandits: Overcoming Catastrophic Forgetting through Likelihood Matching
• Self-Supervised Contextual Bandits in Computer Vision

Cited by in corpus (6)

• Neural-BO: A Black-box Optimization Algorithm using Deep Neural Networks
• EE-Net: Exploitation-Exploration Neural Networks in Contextual Bandits
• Risk-Aware Continuous Control with Neural Contextual Bandits
• Optimal Gradient-based Algorithms for Non-concave Bandit Optimization
• Offline Neural Contextual Bandits: Pessimism, Optimization and Generalization
• Online Limited Memory Neural-Linear Bandits with Likelihood Matching