20 citations · 25 across the 3 of their papers we have counts for
6 papers
Metrology for AI: From Benchmarks to Instruments
Chris Welty, Praveen Paritosh, Lora Aroyo
In this paper we present the first steps towards hardening the science of measuring AI systems, by adopting metrology, the science of measurement and its application, and applying…
What is Fair? Exploring Pareto-Efficiency for Fairness Constrained Classifiers
Ananth Balashankar, Alyssa Lees, Chris Welty +1
The potential for learned models to amplify existing societal biases has been broadly recognized. Fairness-aware classifier constraints, which apply equality metrics of performance…
A Crowdsourced Frame Disambiguation Corpus with Ambiguity
Anca Dumitrache, Lora Aroyo, Chris Welty
We present a resource for the task of FrameNet semantic frame disambiguation of over 5,000 word-sentence pairs from the Wikipedia corpus. The annotations were collected using a nov…
CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement
Anca Dumitrache, Oana Inel, Lora Aroyo +2
Typically crowdsourcing-based approaches to gather annotated data use inter-annotator agreement as a measure of quality. However, in many domains, there is ambiguity in the data, a…
Capturing Ambiguity in Crowdsourcing Frame Disambiguation
Anca Dumitrache, Lora Aroyo, Chris Welty
FrameNet is a computational linguistics resource composed of semantic frames, high-level concepts that represent the meanings of words. In this paper, we present an approach to gat…
False Positive and Cross-relation Signals in Distant Supervision Data
Anca Dumitrache, Lora Aroyo, Chris Welty
Distant supervision (DS) is a well-established method for relation extraction from text, based on the assumption that when a knowledge-base contains a relation between a term pair,…