7 citations · 11 across the 9 of their papers we have counts for
9 papers
BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity
Harshita Diddee, Gregory Yauney, Swabha Swayamdipta +1
Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchma…
CIE: Controlling Language Model Text Generations Using Continuous Signals
Vinay Samuel, Harshita Diddee, Yiming Zhang +1
Aligning language models (LMs) with user intent is becoming increasingly relevant to enhance user experience. This calls for designing methods that can allow users to control the p…
NoveltyBench: Evaluating Language Models for Humanlike Diversity
Yiming Zhang, Harshita Diddee, Susan Holm +5
Language models have demonstrated remarkable capabilities on standard benchmarks, yet they struggle increasingly from mode collapse, the inability to generate diverse and novel out…
Chasing Random: Instruction Selection Strategies Fail to Generalize
Harshita Diddee, Daphne Ippolito
Prior work has shown that language models can be tuned to follow user instructions using only a small set of high-quality instructions. This has accelerated the development of meth…
Customizing Large Language Model Generation Style using Parameter-Efficient Finetuning
Xinyue Liu, Harshita Diddee, Daphne Ippolito
One-size-fits-all large language models (LLMs) are increasingly being used to help people with their writing. However, the style these models are trained to write in may not suit a…
Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology
Rishav Hada, Safiya Husain, Varun Gumma +8
Existing research in measuring and mitigating gender bias predominantly centers on English, overlooking the intricate challenges posed by non-English languages and the Global South…
BEnQA: A Question Answering and Reasoning Benchmark for Bengali and English
Sheikh Shafayat, H M Quamran Hasan, Minhajur Rahman Chowdhury Mahim +3
In this study, we introduce BEnQA, a dataset comprising parallel Bengali and English exam questions for middle and high school levels in Bangladesh. Our dataset consists of approxi…
''Fifty Shades of Bias'': Normative Ratings of Gender Bias in GPT Generated English Text
Rishav Hada, Agrima Seth, Harshita Diddee +1
Language serves as a powerful tool for the manifestation of societal belief systems. In doing so, it also perpetuates the prevalent biases in our society. Gender bias is one of the…
Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?
Rishav Hada, Varun Gumma, Adrian de Wynter +5
Large Language Models (LLMs) excel in various Natural Language Processing (NLP) tasks, yet their evaluation, particularly in languages beyond the top , remains inadequate due t…