343 citations · 748 across the 22 of their papers we have counts for
31 papers
CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation
Jaspreet Ranjit, Swaminathan Chellappa, Augusto Rivas Costante +5
We present CoDA, an open-source human-LM collaborative codebook development assistant that helps domain experts develop annotation guidelines for annotating new variables in unstru…
Are We Automating the Joy Out of Work? Designing AI to Augment Work, Not Meaning
Jaspreet Ranjit, Ke Zhou, Swabha Swayamdipta +1
Prior work has mapped which workplace tasks are exposed to AI, but less is known about whether workers perceive these tasks as meaningful or as busywork. We examined: (1) which dim…
Sample, Align, Synthesize: Graph-Based Response Synthesis with ConGrs
Sayan Ghosh, Shahzaib Saqib Warraich, Dhruv Tarsadiya +2
Language models can be sampled multiple times to access the distribution underlying their responses, but existing methods cannot efficiently synthesize rich epistemic signals acros…
How Reliable is Language Model Micro-Benchmarking?
Gregory Yauney, Shahzaib Saqib Warraich, Swabha Swayamdipta
Micro-benchmarking offers a solution to the often prohibitive time and cost of language model development: evaluate on a very small subset of existing benchmarks. Can these micro-b…
Uncovering Intervention Opportunities for Suicide Prevention with Language Model Assistants
Jaspreet Ranjit, Hyundong J. Cho, Claire J. Smerdon +5
Warning: This paper discusses topics of suicide and suicidal ideation, which may be distressing to some readers. The National Violent Death Reporting System (NVDRS) documents infor…
Evaluation Under Imperfect Benchmarks and Ratings: A Case Study in Text Simplification
Joseph Liu, Yoonsoo Nam, Xinyue Cui +1
Despite the successes of language models, their evaluation remains a daunting challenge for new and existing tasks. We consider the task of text simplification, commonly used to im…
Robust Data Watermarking in Language Models by Injecting Fictitious Knowledge
Xinyue Cui, Johnny Tian-Zheng Wei, Swabha Swayamdipta +1
Data watermarking in language models injects traceable signals, such as specific token sequences or stylistic patterns, into copyrighted text, allowing copyright holders to track a…
Out-of-Distribution Detection through Soft Clustering with Non-Negative Kernel Regression
Aryan Gulati, Xingjian Dong, Carlos Hurtado +3
As language models become more general purpose, increased attention needs to be paid to detecting out-of-distribution (OOD) instances, i.e., those not belonging to any of the distr…
Compare without Despair: Reliable Preference Evaluation with Generation Separability
Sayan Ghosh, Tejas Srinivasan, Swabha Swayamdipta
Human evaluation of generated language through pairwise preference judgments is pervasive. However, under common scenarios, such as when generations from a model pair are very simi…
Annotating FrameNet via Structure-Conditioned Language Generation
Xinyue Cui, Swabha Swayamdipta
Despite the remarkable generative capabilities of language models in producing naturalistic language, their effectiveness on explicit manipulation and generation of linguistic stru…
OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants
Jaspreet Ranjit, Brihi Joshi, Rebecca Dorn +6
Warning: Contents of this paper may be upsetting. Public attitudes towards key societal issues, expressed on online media, are of immense value in policy and reform efforts, yet ch…
Generative Explanations for Program Synthesizers
Amirmohammad Nazari, Souti Chattopadhyay, Swabha Swayamdipta +1
Despite great advances in program synthesis techniques, they remain algorithmic black boxes. Although they guarantee that when synthesis is successful, the implementation satisfies…
Does Video Summarization Require Videos? Quantifying the Effectiveness of Language in Video Summarization
Yoonsoo Nam, Adam Lehavi, Daniel Yang +3
Video summarization remains a huge challenge in computer vision due to the size of the input videos to be summarized. We propose an efficient, language-only video summarizer that a…
Sister Help: Data Augmentation for Frame-Semantic Role Labeling
Ayush Pancholy, Miriam R. L. Petruck, Swabha Swayamdipta
While FrameNet is widely regarded as a rich resource of semantics in natural language processing, a major criticism concerns its lack of coverage and the relative paucity of its la…
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Alisa Liu, Maarten Sap, Ximing Lu +4
Despite recent advances in natural language generation, it remains challenging to control attributes of generated text. We propose DExperts: Decoding-time Experts, a decoding-time…
Contrastive Explanations for Model Interpretability
Alon Jacovi, Swabha Swayamdipta, Shauli Ravfogel +3
Contrastive explanations clarify why an event occurred in contrast to another. They are more inherently intuitive to humans to both produce and comprehend. We propose a methodology…
Challenges in Automated Debiasing for Toxic Language Detection
Xuhui Zhou, Maarten Sap, Swabha Swayamdipta +2
Biased associations have been a challenge in the development of classifiers for detecting toxic language, hindering both fairness and accuracy. As potential solutions, we investiga…
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie +4
Large datasets have become commonplace in NLP research. However, the increased emphasis on data quantity has made it challenging to assess the quality of data. We introduce Data Ma…
The Right Tool for the Job: Matching Model and Instance Complexities
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta +2
As NLP models become larger, executing a trained model requires significant computational resources incurring monetary and environmental costs. To better respect a given inference…
Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta +4
Language models pretrained on text from a wide variety of sources form the foundation of today's NLP. In light of the success of these broad-coverage models, we investigate whether…
Generative Data Augmentation for Commonsense Reasoning
Yiben Yang, Chaitanya Malaviya, Jared Fernandez +6
Recent advances in commonsense reasoning depend on large-scale human-annotated training data to achieve peak performance. However, manual curation of training examples is expensive…
Adversarial Filters of Dataset Biases
Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula +4
Large neural models have demonstrated human-level performance on language and vision benchmarks, while their performance degrades considerably on adversarial or out-of-distribution…
Shallow Syntax in Deep Water
Swabha Swayamdipta, Matthew Peters, Brendan Roof +2
Shallow syntax provides an approximation of phrase-syntactic structure of sentences; it can be produced with high accuracy, and is computationally cheap to obtain. We investigate t…
Syntactic Scaffolds for Semantic Structures
Swabha Swayamdipta, Sam Thomson, Kenton Lee +3
We introduce the syntactic scaffold, an approach to incorporating syntactic information into semantic tasks. Syntactic scaffolds avoid expensive syntactic processing at runtime, on…