Are You Sure You're Sure? Two Confounds in a Sycophancy Benchmark

Atharv Gupta, Akshat Jindal, Lavanya Nigam +1

Sycophancy is a language model's tendency to cave when a user pushes back, abandoning a correct answer for the user's. Several benchmarks now measure it by scripting an objection a…