End User Authoring of Personalized Content Classifiers: Comparing Example Labeling, Rule Writing, and LLM Prompting

Abstract

Existing tools for laypeople to create personal classifiers often assume a motivated user working uninterrupted in a single, lengthy session. However, users tend to engage with social media casually, with many short sessions on an ongoing, daily basis. To make creating personal classifiers for content curation easier for such users, tools should support rapid initialization and iterative refinement. In this work, we compare three strategies -- (1) example labeling, (2) rule writing, and (3) large language model (LLM) prompting -- for end users to build personal content classifiers. From an experiment with 37 non-programmers tasked with creating personalized moderation filters, we found that participants preferred different initializing strategies in different contexts, despite LLM prompting's better performance. However, all strategies faced challenges with iterative refinement. To overcome challenges in iterating on their prompts, participants even adopted hybrid approaches such as providing examples as in-context examples or writing rule-like prompts.

Accepted by CHI'25

References in corpus (7)

• Snorkel: Rapid Training Data Creation with Weak Supervision
• Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors
• Personalizing Content Moderation on Social Media: User Perspectives on Moderation Choices, Interface Design, and Labor
• Designing Word Filter Tools for Creator-led Comment Moderation
• Mapping the Design Space of Teachable Social Media Feed Experiences
• ModSandbox: Facilitating Online Community Moderation Through Error Prediction and Improvement of Automated Rules
• Cura: Curation at Social Media Scale