BanditRank: Learning to Rank Using Contextual Bandits

Abstract

We propose an extensible deep learning method that uses reinforcement learning to train neural networks for offline ranking in information retrieval (IR). We call our method BanditRank as it treats ranking as a contextual bandit problem. In the domain of learning to rank for IR, current deep learning models are trained on objective functions different from the measures they are evaluated on. Since most evaluation measures are discrete quantities, they cannot be leveraged by directly using gradient descent algorithms without an approximation. BanditRank bridges this gap by directly optimizing a task-specific measure, such as mean average precision (MAP), using gradient descent. Specifically, a contextual bandit whose action is to rank input documents is trained using a policy gradient algorithm to directly maximize the reward. The reward can be a single measure, such as MAP, or a combination of several measures. The notion of ranking is also inherent in BanditRank, similar to the current \textit{listwise} approaches. To evaluate the effectiveness of BanditRank, we conducted a series of experiments on datasets related to three different tasks, i.e., web search, community, and factoid question answering. We found that it performs better than state-of-the-art methods when applied on the question answering datasets. On the web search dataset, we found that BanditRank performed better than four strong listwise baselines including LambdaMART, AdaRank, ListNet and Coordinate Ascent.

arXiv comment9 pages

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• A Deep Reinforced Model for Abstractive Summarization
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• Sequence Level Training with Recurrent Neural Networks
• A Deep Relevance Matching Model for Ad-hoc Retrieval
• Attentive Pooling Networks
• An Actor-Critic Algorithm for Sequence Prediction
• Introducing LETOR 4.0 Datasets
• Sentence Similarity Learning by Lexical Decomposition and Composition
• Match-SRNN: Modeling the Recursive Matching Structure with Spatial RNN
• Bilateral Multi-Perspective Matching for Natural Language Sentences
• Cascading Bandits: Learning to Rank in the Cascade Model
• DCM Bandits: Learning to Rank with Multiple Clicks
• BanditSum: Extractive Summarization as a Contextual Bandit
• Multi-Cast Attention Networks for Retrieval-based Question Answering and Response Prediction