Curriculum Learning Strategies for IR: An Empirical Study on Conversation Response Ranking
arXiv:1912.08555
Abstract
Neural ranking models are traditionally trained on a series of random batches, sampled uniformly from the entire training set. Curriculum learning has recently been shown to improve neural models' effectiveness by sampling batches non-uniformly, going from easy to difficult instances during training. In the context of neural Information Retrieval (IR) curriculum learning has not been explored yet, and so it remains unclear (1) how to measure the difficulty of training instances and (2) how to transition from easy to difficult instances during training. To address both challenges and determine whether curriculum learning is beneficial for neural ranking models, we need large-scale datasets and a retrieval task that allows us to conduct a wide range of experiments. For this purpose, we resort to the task of conversation response ranking: ranking responses given the conversation history. In order to deal with challenge (1), we explore scoring functions to measure the difficulty of conversations based on different input spaces. To address challenge (2) we evaluate different pacing functions, which determine the velocity in which we go from easy to difficult instances. We find that, overall, by just intelligently sorting the training data (i.e., by performing curriculum learning) we can improve the retrieval effectiveness by up to 2%.
arXiv commentAccepted for publication in the 42nd European Conference on Information Retrieval (ECIR'20)
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- Distilling the Knowledge in a Neural Network
- Convolutional Neural Network Architectures for Matching Natural Language Sentences
- A Deep Relevance Matching Model for Ad-hoc Retrieval
- Passage Re-ranking with BERT
- A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations
- Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems
- Simple Applications of BERT for Ad Hoc Document Retrieval
- On The Power of Curriculum Learning in Training Deep Networks
- Curriculum Learning and Minibatch Bucketing in Neural Machine Translation
- Analyzing and Characterizing User Intent in Information-seeking Conversations
- An Empirical Exploration of Curriculum Learning for Neural Machine Translation
- Adversarial Generation of Natural Language
- Modeling Multi-turn Conversation with Deep Utterance Aggregation
- End-to-End Answer Chunk Extraction and Ranking for Reading Comprehension
- Boosting Neural Machine Translation
- A Hybrid Retrieval-Generation Neural Conversation Model
- Introducing MANtIS: a novel Multi-Domain Information Seeking Dialogues Dataset
- FAQ Retrieval using Query-Question Similarity and BERT-Based Query-Answer Relevance