63 citations · 143 across the 9 of their papers we have counts for
9 papers
Metrology for AI: From Benchmarks to Instruments
Chris Welty, Praveen Paritosh, Lora Aroyo
In this paper we present the first steps towards hardening the science of measuring AI systems, by adopting metrology, the science of measurement and its application, and applying…
What is Fair? Exploring Pareto-Efficiency for Fairness Constrained Classifiers
Ananth Balashankar, Alyssa Lees, Chris Welty +1
The potential for learned models to amplify existing societal biases has been broadly recognized. Fairness-aware classifier constraints, which apply equality metrics of performance…
A Crowdsourced Frame Disambiguation Corpus with Ambiguity
Anca Dumitrache, Lora Aroyo, Chris Welty
We present a resource for the task of FrameNet semantic frame disambiguation of over 5,000 word-sentence pairs from the Wikipedia corpus. The annotations were collected using a nov…
Empirical Methodology for Crowdsourcing Ground Truth
Anca Dumitrache, Oana Inel, Benjamin Timmermans +4
The process of gathering ground truth data through human annotation is a major bottleneck in the use of information extraction methods for populating the Semantic Web. Crowdsourcin…
Crowdsourcing Semantic Label Propagation in Relation Classification
Anca Dumitrache, Lora Aroyo, Chris Welty
Distant supervision is a popular method for performing relation extraction from text that is known to produce noisy labels. Most progress in relation extraction and classification…
CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement
Anca Dumitrache, Oana Inel, Lora Aroyo +2
Typically crowdsourcing-based approaches to gather annotated data use inter-annotator agreement as a measure of quality. However, in many domains, there is ambiguity in the data, a…
Capturing Ambiguity in Crowdsourcing Frame Disambiguation
Anca Dumitrache, Lora Aroyo, Chris Welty
FrameNet is a computational linguistics resource composed of semantic frames, high-level concepts that represent the meanings of words. In this paper, we present an approach to gat…
False Positive and Cross-relation Signals in Distant Supervision Data
Anca Dumitrache, Lora Aroyo, Chris Welty
Distant supervision (DS) is a well-established method for relation extraction from text, based on the assumption that when a knowledge-base contains a relation between a term pair,…
Crowdsourcing Ground Truth for Medical Relation Extraction
Anca Dumitrache, Lora Aroyo, Chris Welty
Cognitive computing systems require human labeled data for evaluation, and often for training. The standard practice used in gathering this data minimizes disagreement between anno…