359 citations · 994 across the 28 of their papers we have counts for
10 papers · 1 filter
Spurious Correlations in Reference-Free Evaluation of Text Generation
Esin Durmus, Faisal Ladhak, Tatsunori Hashimoto
Model-based, reference-free evaluation metrics have been proposed as a fast and cost-effective approach to evaluate Natural Language Generation (NLG) systems. Despite promising rec…
Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions
Dorottya Demszky, Jing Liu, Zid Mancenido +4
In conversation, uptake happens when a speaker builds on the contribution of their interlocutor by, for example, acknowledging, repeating or reformulating what they have said. In e…
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Tianyi Zhang, Tatsunori Hashimoto
We study how masking and predicting tokens in an unsupervised fashion can give rise to linguistic structures and downstream performance gains. Recent theories have suggested that p…
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal +53
We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of auto…
Improved Natural Language Generation via Loss Truncation
Daniel Kang, Tatsunori Hashimoto
Neural language models are usually trained to match the distributional properties of a large-scale corpus by minimizing the log loss. While straightforward to optimize, this approa…
Learning Autocomplete Systems as a Communication Game
Mina Lee, Tatsunori B. Hashimoto, Percy Liang
We study textual autocomplete---the task of predicting a full sentence from a partial sentence---as a human-machine communication game. Specifically, we consider three competing go…
Distributionally Robust Language Modeling
Yonatan Oren, Shiori Sagawa, Tatsunori B. Hashimoto +1
Language models are generally trained on data spanning a wide range of topics (e.g., news, reviews, fiction), but they might be applied to an a priori unknown target distribution (…
Unifying Human and Statistical Evaluation for Natural Language Generation
Tatsunori B. Hashimoto, Hugh Zhang, Percy Liang
How can we measure whether a natural language generation system produces both high quality and diverse outputs? Human evaluation captures quality but not diversity, as it does not…
Generating Sentences by Editing Prototypes
Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren +1
We propose a new generative model of sentences that first samples a prototype sentence from the training corpus and then edits it into a new sentence. Compared to traditional model…
Word, graph and manifold embedding from Markov processes
Tatsunori B. Hashimoto, David Alvarez-Melis, Tommi S. Jaakkola
Continuous vector representations of words and objects appear to carry surprisingly rich semantic content. In this paper, we advance both the conceptual and theoretical understandi…