8 papers
Community-Informed AI Models for Police Accountability
Benjamin A. T. Graham, Lauren Brown, Georgios Chochlakis +13
Face-to-face interactions between police officers and the public affect both individual well-being and democratic legitimacy. Many government-public interactions are captured on vi…
The Subjectivity of Respect in Police Traffic Stops: Modeling Community Perspectives in Body-Worn Camera Footage
Preni Golazizian, Elnaz Rahmati, Jackson Trager +17
Traffic stops are among the most frequent police-civilian interactions, and body-worn cameras (BWCs) provide a unique record of how these encounters unfold. Respect is a central di…
Authors Should Label Their Own Documents
Marcus Ma, Cole Johnson, Nolan Bridges +3
Third-party annotation is the status quo for labeling text, but egocentric information such as sentiment and belief can at best only be approximated by a third-person proxy. We int…
Semantic F1 Scores: Fair Evaluation Under Fuzzy Class Boundaries
Georgios Chochlakis, Jackson Trager, Vedant Jhaveri +3
We propose Semantic F1 Scores, novel evaluation metrics for subjective or fuzzy multi-label classification that quantify semantic relatedness between predicted and gold labels. Unl…
Large Language Models Do Multi-Label Classification Differently
Marcus Ma, Georgios Chochlakis, Niyantha Maruthu Pandiyan +2
Multi-label classification is prevalent in real-world settings, but the behavior of Large Language Models (LLMs) in this setting is understudied. We investigate how autoregressive…
Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts
Georgios Chochlakis, Peter Wu, Arjun Bedi +3
Modeling complex subjective tasks in Natural Language Processing, such as recognizing emotion and morality, is considerably challenging due to significant variation in human annota…