Crowd-sourcing NLG Data: Pictures Elicit Better Data

Abstract

Recent advances in corpus-based Natural Language Generation (NLG) hold the promise of being easily portable across domains, but require costly training data, consisting of meaning representations (MRs) paired with Natural Language (NL) utterances. In this work, we propose a novel framework for crowdsourcing high quality NLG training data, using automatic quality control measures and evaluating different MRs with which to elicit data. We show that pictorial MRs result in better NL data being collected than logic-based MRs: utterances elicited by pictorial MRs are judged as significantly more natural, more informative, and better phrased, with a significant increase in average quality ratings (around 0.5 points on a 6-point scale), compared to using the logical MRs. As the MR becomes more complex, the benefits of pictorial stimuli increase. The collected data will be released as part of this submission.

The 9th International Natural Language Generation conference INLG, 2016. 10 pages, 2 figures, 3 tables

Cited by in corpus (4)

• Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge
• Characterizing Variation in Crowd-Sourced Data for Training Neural Language Generators to Produce Stylistically Varied Outputs
• Neural MultiVoice Models for Expressing Novel Personalities in Dialog
• Data-driven Natural Language Generation: Paving the Road to Success