Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring

Abstract

The use of deep pre-trained bidirectional transformers has led to remarkable progress in a number of applications (Devlin et al., 2018). For tasks that make pairwise comparisons between sequences, matching a given input with a corresponding label, two approaches are common: Cross-encoders performing full self-attention over the pair and Bi-encoders encoding the pair separately. The former often performs better, but is too slow for practical use. In this work, we develop a new transformer architecture, the Poly-encoder, that learns global rather than token level self-attention features. We perform a detailed comparison of all three approaches, including what pre-training and fine-tuning strategies work best. We show our models achieve state-of-the-art results on three existing tasks; that Poly-encoders are faster than Cross-encoders and more accurate than Bi-encoders; and that the best results are obtained by pre-training on large datasets similar to the downstream tasks.

arXiv commentICLR 2020

References in corpus (10)

• Wizard of Wikipedia: Knowledge-Powered Conversational agents
• TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
• Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems
• Improved Deep Learning Baselines for Ubuntu Corpus Dialogs
• Modeling Multi-turn Conversation with Deep Utterance Aggregation
• ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
• Sequential Attention-based Network for Noetic End-to-End Response Selection
• Dialog System Technology Challenge 7
• Enhance word representation for out-of-vocabulary on Ubuntu dialogue corpus
• Building Sequential Inference Models for End-to-End Response Selection

Cited by in corpus (42)

• Whitening Sentence Representations for Better Semantics and Faster Retrieval
• Autoregressive Entity Retrieval
• Information Retrieval: Recent Advances and Beyond
• Semantic Models for the First-stage Retrieval: A Comprehensive Review
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
• Distilling Knowledge from Reader to Retriever for Question Answering
• Query Embedding Pruning for Dense Retrieval
• Scalable Zero-shot Entity Linking with Dense Entity Retrieval
• Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
• Persona-Based Conversational AI: State of the Art and Challenges
• Augmenting Transformers with KNN-Based Composite Memory for Dialogue
• Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment
• Simulating and Modeling the Risk of Conversational Search
• Focus on what matters: Applying Discourse Coherence Theory to Cross Document Coreference
• I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling
• I love your chain mail! Making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents
• Evaluating Empathetic Chatbots in Customer Service Settings
• All-in-One Image-Grounded Conversational Agents
• Dialogue Response Ranking Training with Large-Scale Human Feedback Data
• MMEAD: MS MARCO Entity Annotations and Disambiguations
• Multistage BiCross encoder for multilingual access to COVID-19 health information
• Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations
• SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval
• Pre-trained Language Model for Web-scale Retrieval in Baidu Search
• Generating Interactive Worlds with Text
• Context Retrieval via Normalized Contextual Latent Interaction for Conversational Agent
• Enriching Conversation Context in Retrieval-based Chatbots
• Controlling the Risk of Conversational Search via Reinforcement Learning
• Training Large-Scale News Recommenders with Pretrained Language Models in the Loop
• Building an Efficient and Effective Retrieval-based Dialogue System via Mutual Learning
• Adversarial Semantic Collisions
• Multi-turn Response Selection using Dialogue Dependency Relations
• Evaluation of Semantic Answer Similarity Metrics
• ConvFiT: Conversational Fine-Tuning of Pretrained Language Models
• A Neural Network-Based Linguistic Similarity Measure for Entrainment in Conversations
• Persona-Coded Poly-Encoder: Persona-Guided Multi-Stream Conversational Sentence Scoring
• MIRA: Leveraging Multi-Intention Co-click Information in Web-scale Document Retrieval using Deep Neural Networks
• Quality and Cost Trade-offs in Passage Re-ranking Task
• Viola: A Topic Agnostic Generate-and-Rank Dialogue System
• Hybrid Encoder: Towards Efficient and Precise Native AdsRecommendation via Hybrid Transformer Encoding Networks