6 papers
SearchLog: A Web Browser Extension for Capturing Search Logs in Laboratory Studies
Jiaman He, Riccardo Xia, Dana McKay +2
Natural search logs are valuable for studying search behavior in information seeking settings. We present SearchLog, an easy-to-install web browser extension for collecting natural…
Characterizing Personality from Eye-Tracking: The Role of Gaze and Its Absence in Interactive Search Environments
Jiaman He, Marta Micheli, Damiano Spina +3
Personality traits influence how individuals engage, behave, and make decisions during the information-seeking process. However, few studies have linked personality to observable s…
Dagstuhl Perspectives Workshop 24352 -- Conversational Agents: A Framework for Evaluation (CAFE): Manifesto
Christine Bauer, Li Chen, Nicola Ferro +19
During the workshop, we deeply discussed what CONversational Information ACcess (CONIAC) is and its unique features, proposing a world model abstracting it, and defined the Convers…
Can We Hide Machines in the Crowd? Quantifying Equivalence in LLM-in-the-loop Annotation Tasks
Jiaman He, Zikang Leng, Dana McKay +2
Many evaluations of large language models (LLMs) in text annotation focus primarily on the correctness of the output, typically comparing model-generated labels to human-annotated…
Characterising Topic Familiarity and Query Specificity Using Eye-Tracking Data
Jiaman He, Zikang Leng, Dana McKay +2
Eye-tracking data has been shown to correlate with a user's knowledge level and query formulation behaviour. While previous work has focused primarily on eye gaze fixations for att…
Multi-stage Large Language Model Pipelines Can Outperform GPT-4o in Relevance Assessment
Julian A. Schnabel, Johanne R. Trippas, Falk Scholer +1
The effectiveness of search systems is evaluated using relevance labels that indicate the usefulness of documents for specific queries and users. While obtaining these relevance la…