44 citations · 223 across the 40 of their papers we have counts for
4 papers · 1 filter
Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks
Alexander Braylan, Omar Alonso, Matthew Lease
When annotators label data, a key metric for quality assurance is inter-annotator agreement (IAA): the extent to which annotators agree on their labels. Though many IAA measures ex…
ProtoTEx: Explaining Model Decisions with Prototype Tensors
Anubrata Das, Chitrank Gupta, Venelin Kovatchev +2
We present ProtoTEx, a novel white-box NLP classification architecture based on prototype networks. ProtoTEx faithfully explains model decisions based on prototype tensors that enc…
Finding Pareto Trade-offs in Fair and Accurate Detection of Toxic Speech
Soumyajit Gupta, Venelin Kovatchev, Anubrata Das +2
Optimizing NLP models for fairness poses many challenges. Lack of differentiable fairness measures prevents gradient-based loss training or requires surrogate losses that diverge f…
Designing Closed Human-in-the-loop Deferral Pipelines
Vijay Keswani, Matthew Lease, Krishnaram Kenthapadi
In hybrid human-machine deferral frameworks, a classifier can defer uncertain cases to human decision-makers (who are often themselves fallible). Prior work on simultaneous trainin…