The Symmetry between Arms and Knapsacks: A Primal-Dual Approach for Bandits with Knapsacks

Abstract

In this paper, we study the bandits with knapsacks (BwK) problem and develop a primal-dual based algorithm that achieves a problem-dependent logarithmic regret bound. The BwK problem extends the multi-arm bandit (MAB) problem to model the resource consumption associated with playing each arm, and the existing BwK literature has been mainly focused on deriving asymptotically optimal distribution-free regret bounds. We first study the primal and dual linear programs underlying the BwK problem. From this primal-dual perspective, we discover symmetry between arms and knapsacks, and then propose a new notion of sub-optimality measure for the BwK problem. The sub-optimality measure highlights the important role of knapsacks in determining algorithm regret and inspires the design of our two-phase algorithm. In the first phase, the algorithm identifies the optimal arms and the binding knapsacks, and in the second phase, it exhausts the binding knapsacks via playing the optimal arms through an adaptive procedure. Our regret upper bound involves the proposed sub-optimality measure and it has a logarithmic dependence on length of horizon $T$ and a polynomial dependence on $m$ (the numbers of arms) and $d$ (the number of knapsacks). To the best of our knowledge, this is the first problem-dependent logarithmic regret bound for solving the general BwK problem.

References in corpus (8)

• Analysis of Thompson Sampling for the multi-armed bandit problem
• Bandits with Knapsacks
• Uniformly bounded regret in the multi-secretary problem
• Linear Contextual Bandits with Knapsacks
• An efficient algorithm for contextual bandits with knapsacks, and an extension to concave objectives
• Online Linear Programming: Dual Convergence, New Algorithms, and Regret Bounds
• Logarithmic regret bounds for Bandits with Knapsacks
• Online Learning with Vector Costs and Bandits with Knapsacks