XYBench: Can LLMs Respond Pragmatically to Queries with Misconceptions?

Akhila Yerukola, Jena D. Hwang, Mingqian Zheng +5

When non-expert users ask LLMs for assistance, their queries can often have misconceptions (e.g., "How do I parse XML with regex?"). In such cases, often referred to as the XY-prob…