44 citations · 223 across the 40 of their papers we have counts for
41 papers
How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets
Danula Hettiachchi, Indigo Holcombe-James, Stephanie Livingstone +4
Crowdsourced annotation is vital to both collecting labelled data to train and test automated content moderation systems and to support human-in-the-loop review of system decisions…
Designing Closed-Loop Models for Task Allocation
Vijay Keswani, L. Elisa Celis, Krishnaram Kenthapadi +1
Automatically assigning tasks to people is challenging because human performance can vary across tasks for many reasons. This challenge is further compounded in real-life settings…
Same Same, But Different: Conditional Multi-Task Learning for Demographic-Specific Toxicity Detection
Soumyajit Gupta, Sooyong Lee, Maria De-Arteaga +1
Algorithmic bias often arises as a result of differential subgroup validity, in which predictive relationships vary across groups. For example, in toxic language detection, comment…
Learning Complementary Policies for Human-AI Teams
Ruijiang Gao, Maytal Saar-Tsechansky, Maria De-Arteaga
This paper tackles the critical challenge of human-AI complementarity in decision-making. Departing from the traditional focus on algorithmic performance in favor of performance of…
New Metrics to Encourage Innovation and Diversity in Information Retrieval Approaches
Mehmet Deniz Türkmen, Matthew Lease, Mucahid Kutlu
In evaluation campaigns, participants often explore variations of popular, state-of-the-art baselines as a low-risk strategy to achieve competitive results. While effective, this c…
Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks
Alexander Braylan, Omar Alonso, Matthew Lease
When annotators label data, a key metric for quality assurance is inter-annotator agreement (IAA): the extent to which annotators agree on their labels. Though many IAA measures ex…
ProtoTEx: Explaining Model Decisions with Prototype Tensors
Anubrata Das, Chitrank Gupta, Venelin Kovatchev +2
We present ProtoTEx, a novel white-box NLP classification architecture based on prototype networks. ProtoTEx faithfully explains model decisions based on prototype tensors that enc…
Finding Pareto Trade-offs in Fair and Accurate Detection of Toxic Speech
Soumyajit Gupta, Venelin Kovatchev, Anubrata Das +2
Optimizing NLP models for fairness poses many challenges. Lack of differentiable fairness measures prevents gradient-based loss training or requires surrogate losses that diverge f…
Designing Closed Human-in-the-loop Deferral Pipelines
Vijay Keswani, Matthew Lease, Krishnaram Kenthapadi
In hybrid human-machine deferral frameworks, a classifier can defer uncertain cases to human decision-makers (who are often themselves fallible). Prior work on simultaneous trainin…
In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers
Vivek Krishna Pradhan, Mike Schaekermann, Matthew Lease
We propose a novel three-stage FIND-RESOLVE-LABEL workflow for crowdsourced annotation to reduce ambiguity in task instructions and thus improve annotation quality. Stage 1 (FIND)…
Data Excellence for AI: Why Should You Care
Lora Aroyo, Matthew Lease, Praveen Paritosh +1
The efficacy of machine learning (ML) models depends on both algorithms and data. Training data defines what we want our models to learn, and testing data provides the means by whi…
Scalable Unidirectional Pareto Optimality for Multi-Task Learning with Constraints
Soumyajit Gupta, Gurpreet Singh, Raghu Bollapragada +1
Multi-objective optimization (MOO) problems require balancing competing objectives, often under constraints. The Pareto optimal solution set defines all possible optimal trade-offs…
The Case for Claim Difficulty Assessment in Automatic Fact Checking
Prakhar Singh, Anubrata Das, Junyi Jessy Li +1
Fact-checking is the process of evaluating the veracity of claims (i.e., purported facts). In this opinion piece, we raise an issue that has received little attention in prior work…
An Information Retrieval Approach to Building Datasets for Hate Speech Detection
Md Mustafizur Rahman, Dinesh Balakrishnan, Dhiraj Murthy +2
Building a benchmark dataset for hate speech detection presents various challenges. Firstly, because hate speech is relatively rare, random sampling of tweets to annotate is very i…
Human-AI Collaboration with Bandit Feedback
Ruijiang Gao, Maytal Saar-Tsechansky, Maria De-Arteaga +3
Human-machine complementarity is important when neither the algorithm nor the human yield dominant performance across all instances in a given domain. Most research on algorithmic…
The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help
Danula Hettiachchi, Mike Schaekermann, Tristan McKinney +1
We consider a class of variable effort human annotation tasks in which the number of labels required per item can greatly vary (e.g., finding all faces in an image, named entities…
A Hybrid 2-stage Neural Optimization for Pareto Front Extraction
Gurpreet Singh, Soumyajit Gupta, Matthew Lease +1
Classification, recommendation, and ranking problems often involve competing goals with additional constraints (e.g., to satisfy fairness or diversity criteria). Such optimization…
Towards Unbiased and Accurate Deferral to Multiple Experts
Vijay Keswani, Matthew Lease, Krishnaram Kenthapadi
Machine learning models are often implemented in cohort with humans in the pipeline, with the model having an option to defer to a domain expert in cases where it has low confidenc…
SCA-Net: A Self-Correcting Two-Layer Autoencoder for Hyper-spectral Unmixing
Gurpreet Singh, Soumyajit Gupta, Clint Dawson
Hyperspectral unmixing involves separating a pixel as a weighted combination of its constituent endmembers and corresponding fractional abundances, with the current state of the ar…
Understanding and Predicting Characteristics of Test Collections in Information Retrieval
Md Mustafizur Rahman, Mucahid Kutlu, Matthew Lease
Research community evaluations in information retrieval, such as NIST's Text REtrieval Conference (TREC), build reusable test collections by pooling document rankings submitted by…
You Are What You Tweet: Profiling Users by Past Tweets to Improve Hate Speech Detection
Prateek Chaudhry, Matthew Lease
Hate speech detection research has predominantly focused on purely content-based methods, without exploiting any additional context. We briefly critique pros and cons of this task…
Range-Net: A High Precision Streaming SVD for Big Data Applications
Gurpreet Singh, Soumyajit Gupta, Matthew Lease +1
In a Big Data setting computing the dominant SVD factors is restrictive due to the main memory requirements. Recently introduced streaming Randomized SVD schemes work under the res…
Extracting Optimal Solution Manifolds using Constrained Neural Optimization
Gurpreet Singh, Soumyajit Gupta, Matthew Lease
Constrained Optimization solution algorithms are restricted to point based solutions. In practice, single or multiple objectives must be satisfied, wherein both the objective funct…
A Conceptual Framework for Evaluating Fairness in Search
Anubrata Das, Matthew Lease
While search efficacy has been evaluated traditionally on the basis of result relevance, fairness of search has attracted recent attention. In this work, we define a notion of dist…