44 citations · 223 across the 40 of their papers we have counts for
10 papers · 1 filter
In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers
Vivek Krishna Pradhan, Mike Schaekermann, Matthew Lease
We propose a novel three-stage FIND-RESOLVE-LABEL workflow for crowdsourced annotation to reduce ambiguity in task instructions and thus improve annotation quality. Stage 1 (FIND)…
Data Excellence for AI: Why Should You Care
Lora Aroyo, Matthew Lease, Praveen Paritosh +1
The efficacy of machine learning (ML) models depends on both algorithms and data. Training data defines what we want our models to learn, and testing data provides the means by whi…
Scalable Unidirectional Pareto Optimality for Multi-Task Learning with Constraints
Soumyajit Gupta, Gurpreet Singh, Raghu Bollapragada +1
Multi-objective optimization (MOO) problems require balancing competing objectives, often under constraints. The Pareto optimal solution set defines all possible optimal trade-offs…
The Case for Claim Difficulty Assessment in Automatic Fact Checking
Prakhar Singh, Anubrata Das, Junyi Jessy Li +1
Fact-checking is the process of evaluating the veracity of claims (i.e., purported facts). In this opinion piece, we raise an issue that has received little attention in prior work…
An Information Retrieval Approach to Building Datasets for Hate Speech Detection
Md Mustafizur Rahman, Dinesh Balakrishnan, Dhiraj Murthy +2
Building a benchmark dataset for hate speech detection presents various challenges. Firstly, because hate speech is relatively rare, random sampling of tweets to annotate is very i…
Human-AI Collaboration with Bandit Feedback
Ruijiang Gao, Maytal Saar-Tsechansky, Maria De-Arteaga +3
Human-machine complementarity is important when neither the algorithm nor the human yield dominant performance across all instances in a given domain. Most research on algorithmic…
The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help
Danula Hettiachchi, Mike Schaekermann, Tristan McKinney +1
We consider a class of variable effort human annotation tasks in which the number of labels required per item can greatly vary (e.g., finding all faces in an image, named entities…
A Hybrid 2-stage Neural Optimization for Pareto Front Extraction
Gurpreet Singh, Soumyajit Gupta, Matthew Lease +1
Classification, recommendation, and ranking problems often involve competing goals with additional constraints (e.g., to satisfy fairness or diversity criteria). Such optimization…
Towards Unbiased and Accurate Deferral to Multiple Experts
Vijay Keswani, Matthew Lease, Krishnaram Kenthapadi
Machine learning models are often implemented in cohort with humans in the pipeline, with the model having an option to defer to a domain expert in cases where it has low confidenc…
SCA-Net: A Self-Correcting Two-Layer Autoencoder for Hyper-spectral Unmixing
Gurpreet Singh, Soumyajit Gupta, Clint Dawson
Hyperspectral unmixing involves separating a pixel as a weighted combination of its constituent endmembers and corresponding fractional abundances, with the current state of the ar…