4 papers
BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity
Harshita Diddee, Gregory Yauney, Swabha Swayamdipta +1
Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchma…
CIE: Controlling Language Model Text Generations Using Continuous Signals
Vinay Samuel, Harshita Diddee, Yiming Zhang +1
Aligning language models (LMs) with user intent is becoming increasingly relevant to enhance user experience. This calls for designing methods that can allow users to control the p…
NoveltyBench: Evaluating Language Models for Humanlike Diversity
Yiming Zhang, Harshita Diddee, Susan Holm +5
Language models have demonstrated remarkable capabilities on standard benchmarks, yet they struggle increasingly from mode collapse, the inability to generate diverse and novel out…
Chasing Random: Instruction Selection Strategies Fail to Generalize
Harshita Diddee, Daphne Ippolito
Prior work has shown that language models can be tuned to follow user instructions using only a small set of high-quality instructions. This has accelerated the development of meth…