5 papers
Plans for Evaluating Structured Generative Search Summaries
Tetsuya Sakai, Jina Lee, Hanpei Fang +1
We propose a framework for evaluating structured generative search summaries that are placed atop organic web search results. A structured summary, generated by a large language mo…
Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessment
Nuo Chen, Hanpei Fang, Piaohong Wang +3
Recent studies have shown that prompting can enable large language models (LLMs) to simulate specific personality traits and produce behaviors that align with those traits. However…
Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusing
Nuo Chen, Hanpei Fang, Jiqun Liu +3
Recent research has explored LLMs as scalable tools for relevance labeling, but studies indicate they are susceptible to priming effects, where prior relevance judgments influence…
Do Large Language Models Favor Recent Content? A Study on Recency Bias in LLM-Based Reranking
Hanpei Fang, Sijie Tao, Nuo Chen +2
Large language models (LLMs) are increasingly deployed in information systems, including being used as second-stage rerankers in information retrieval pipelines, yet their suscepti…
Decoy Effect In Search Interaction: Understanding User Behavior and Measuring System Vulnerability
Nuo Chen, Jiqun Liu, Hanpei Fang +3
This study examines the decoy effect's underexplored influence on user search interactions and methods for measuring information retrieval (IR) systems' vulnerability to this effec…