Can Transformer Models Measure Coherence In Text? Re-Thinking the Shuffle Test

Philippe Laban, Luke Dai, Lucas Bandarkar +1

The Shuffle Test is the most common task to evaluate whether NLP models can measure coherence in text. Most recent work uses direct supervision on the task; we show that by simply…