Why We Need New Evaluation Metrics for NLG

Abstract

The majority of NLG evaluation relies on automatic metrics, such as BLEU . In this paper, we motivate the need for novel, system- and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.

accepted to EMNLP 2017

References in corpus (8)

• A User Simulator for Task-Completion Dialogues
• Continuously Learning Neural Dialogue Management
• A Context-aware Natural Language Generator for Dialogue Systems
• Natural Language Generation in Dialogue using Lexicalized and Delexicalized Data
• Investigation of Language Understanding Impact for Reinforcement Learning Based Dialogue Systems
• What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment
• Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systems
• Referenceless Quality Estimation for Natural Language Generation

Cited by in corpus (39)

• Ensemble-Based Deep Reinforcement Learning for Chatbots
• DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• Challenges in Building Intelligent Open-domain Dialog Systems
• A Survey on Recent Advances and Challenges in Reinforcement Learning Methods for Task-Oriented Dialogue Policy Learning
• Sequence-to-Sequence Learning for Task-oriented Dialogue with Dialogue State Representation
• Emotionally-Aware Chatbots: A Survey
• Modelling Hierarchical Structure between Dialogue Policy and Natural Language Generator with Option Framework for Task-oriented Dialogue System
• Hierarchical Imitation and Reinforcement Learning
• Neural Language Generation: Formulation, Methods, and Evaluation
• Aggregating E-commerce Search Results from Heterogeneous Sources via Hierarchical Reinforcement Learning
• A Survey on Dialog Management: Recent Advances and Challenges
• Robust Conversational AI with Grounded Text Generation
• ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems
• Distributed Structured Actor-Critic Reinforcement Learning for Universal Dialogue Management
• SUMBT+LaRL: Effective Multi-domain End-to-end Neural Task-oriented Dialog System
• On LLM Wizards: Identifying Large Language Models' Behaviors for Wizard of Oz Experiments
• A Slot Is Not Built in One Utterance: Spoken Language Dialogs with Sub-Slots
• Meta Dialogue Policy Learning
• A Multimodal Dialogue System for Conversational Image Editing
• AgentGraph: Towards Universal Dialogue Management with Structured Deep Reinforcement Learning
• Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation
• Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialogue
• Document-editing Assistants and Model-based Reinforcement Learning as a Path to Conversational AI
• How to Evaluate Your Dialogue Models: A Review of Approaches
• Automatic Curriculum Learning With Over-repetition Penalty for Dialogue Policy Learning
• UniConv: A Unified Conversational Neural Architecture for Multi-domain Task-oriented Dialogues
• Deep Reinforcement Learning for Chatbots Using Clustered Actions and Human-Likeness Rewards
• Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition
• Towards End-to-End Learning for Efficient Dialogue Agent by Modeling Looking-ahead Ability
• SYNERGY: Building Task Bots at Scale Using Symbolic Knowledge and Machine Teaching
• NaRLE: Natural Language Models using Reinforcement Learning with Emotion Feedback
• Wizard of Search Engine: Access to Information Through Conversations with Search Engines
• Integrating Pretrained Language Model for Dialogue Policy Learning
• Learning Goal-oriented Dialogue Policy with Opposite Agent Awareness
• Hierarchical Context Enhanced Multi-Domain Dialogue System for Multi-domain Task Completion
• Addressing Objects and Their Relations: The Conversational Entity Dialogue Model
• NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions
• Generalization in Text-based Games via Hierarchical Reinforcement Learning