Modeling Document Interactions for Learning to Rank with Regularized Self-Attention

Abstract

Learning to rank is an important task that has been successfully deployed in many real-world information retrieval systems. Most existing methods compute relevance judgments of documents independently, without holistically considering the entire set of competing documents. In this paper, we explore modeling documents interactions with self-attention based neural networks. Although self-attention networks have achieved state-of-the-art results in many NLP tasks, we find empirically that self-attention provides little benefit over baseline neural learning to rank architecture. To improve the learning of self-attention weights, We propose simple yet effective regularization terms designed to model interactions between documents. Evaluations on publicly available Learning to Rank (LETOR) datasets show that training self-attention network with our proposed regularization terms can significantly outperform existing learning to rank methods.

5 pages,5 figures

References in corpus (8)

• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• A Structured Self-attentive Sentence Embedding
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• A Deep Relevance Matching Model for Ad-hoc Retrieval
• Introducing LETOR 4.0 Datasets
• Long Short-Term Memory-Networks for Machine Reading
• Deep Relevance Ranking Using Enhanced Document-Query Interactions