deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets

Abstract

We introduce Discriminative BLEU (deltaBLEU), a novel metric for intrinsic evaluation of generated text in tasks that admit a diverse range of possible outputs. Reference strings are scored for quality by human raters on a scale of [-1, +1] to weight multi-reference BLEU. In tasks involving generation of conversational responses, deltaBLEU correlates reasonably with human judgments and outperforms sentence-level and IBM BLEU in terms of both Spearman's rho and Kendall's tau.

arXiv comment6 pages, to appear at ACL 2015

Cited by in corpus (33)

• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Deep Reinforcement Learning for Dialogue Generation
• Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation
• A Survey of Available Corpora for Building Data-Driven Dialogue Systems
• A Persona-Based Neural Conversation Model
• Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
• Neural Models for Information Retrieval
• Multi-Task Learning for Speaker-Role Adaptation in Neural Conversation Models
• LSTM based Conversation Models
• Analyzing Uncertainty in Neural Machine Translation
• Conversational Contextual Cues: The Case of Personalization and History for Response Ranking
• Meta-evaluation of Conversational Search Evaluation Metrics
• Topic-based Evaluation for Conversational Bots
• Unifying Human and Statistical Evaluation for Natural Language Generation
• A Conditional Variational Framework for Dialog Generation
• An Attentional Neural Conversation Model with Improved Specificity
• Neural Language Generation: Formulation, Methods, and Evaluation
• A Hybrid Retrieval-Generation Neural Conversation Model
• Video Captioning via Hierarchical Reinforcement Learning
• Personalizing a Dialogue System with Transfer Reinforcement Learning
• Reinforcement Learning Based Emotional Editing Constraint Conversation Generation
• Automatic Article Commenting: the Task and Dataset
• Learning to Start for Sequence to Sequence Architecture
• Building Language Models for Text with Named Entities
• Report from the NSF Future Directions Workshop, Toward User-Oriented Agents: Research Directions and Challenges
• Polite Dialogue Generation Without Parallel Data
• Language Model Augmented Relevance Score
• Learning to Disambiguate by Asking Discriminative Questions
• How to Evaluate Your Dialogue Models: A Review of Approaches
• Filtering Noisy Dialogue Corpora by Connectivity and Content Relatedness
• On the Use of Linguistic Features for the Evaluation of Generative Dialogue Systems
• Generating Diverse Translation from Model Distribution with Dropout
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach