334 citations · 740 across the 9 of their papers we have counts for
3 papers · 1 filter
Decision Quality Evaluation Framework at Pinterest
Yuqi Tian, Robert Paine, Attila Dobi +3
Online platforms require robust systems to enforce content safety policies at scale. A critical component of these systems is the ability to evaluate the quality of moderation deci…
Measuring the Prevalence of Policy Violating Content with ML Assisted Sampling and LLM Labeling
Attila Dobi, Aravindh Manickavasagam, Benjamin Thompson +2
Content safety teams need metrics that reflect what users actually experience, not only what is reported. We study prevalence: the fraction of user views (impressions) that went to…
Calibrate Globally, Measure Everywhere: Scaling LLM-Based Prevalence Measurement Across A/B Experiments
Zehao Xu, Tony Paek, Kevin O'Sullivan +1
Online media platforms track the share of impressions associated with content attributes, or prevalence, to evaluate trade-offs and set guardrails in A/B experiments. LLM-based lab…