Sentence Pair Scoring: Towards Unified Framework for Text Comprehension

Abstract

We review the task of Sentence Pair Scoring, popular in the literature in various forms - viewed as Answer Sentence Selection, Semantic Text Scoring, Next Utterance Ranking, Recognizing Textual Entailment, Paraphrasing or e.g. a component of Memory Networks. We argue that all such tasks are similar from the model perspective and propose new baselines by comparing the performance of common IR metrics and popular convolutional, recurrent and attention-based neural models across many Sentence Pair Scoring tasks and datasets. We discuss the problem of evaluating randomized models, propose a statistically grounded methodology, and attempt to improve comparisons by releasing new datasets that are much harder than some of the currently used well explored benchmarks. We introduce a unified open source software framework with easily pluggable models and tasks, which enables us to experiment with multi-task reusability of trained sentence model. We set a new state-of-art in performance on the Ubuntu Dialogue dataset.

submitted as paper to CoNLL 2016

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Convolutional Neural Networks for Sentence Classification
• Reasoning about Entailment with Neural Attention
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• Transition-Based Dependency Parsing with Stack Long Short-Term Memory
• A large annotated corpus for learning natural language inference
• Deep Learning for Answer Sentence Selection
• Attentive Pooling Networks
• Improved Deep Learning Baselines for Ubuntu Corpus Dialogs
• Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems

Cited by in corpus (5)

• DeepMath - Deep Sequence Models for Premise Selection
• Training Neural Response Selection for Task-Oriented Dialogue Systems
• Enhance word representation for out-of-vocabulary on Ubuntu dialogue corpus
• Improving Retrieval Modeling Using Cross Convolution Networks And Multi Frequency Word Embedding
• Selection Bias Explorations and Debias Methods for Natural Language Sentence Matching Datasets