Knowledge-Certainty Diversity Predicts Human-Human and Human-AI Open-Ended Discussion Synergy and Reveals Pure-AI Discussion Shortfalls

Abstract

Conversations transform individual knowledge into collective insight, enabling collaborators to solve problems more accurately than they could alone. Whether dialogues among large language models (LLMs) can replicate the synergistic gains observed in human discussion remains unclear. We systematically compared four interaction settings, LLM-LLM pairs, LLM trios, human trios, and human-LLM pairs, using validated medical multiple-choice questions. Agents answered individually, engaged in open-ended discussion, then re-answered, allowing us to quantify conversational gains. Interactions that included humans consistently yielded synergy (post-discussion accuracy increased for both stronger and weaker participants), whereas purely LLM groups did not improve and often declined. To explain and anticipate when open-ended dialogue helps, we introduce an agent-agnostic knowledge-certainty framework that models each participant by performance (accuracy) and confidence. This framework quantifies knowledge-certainty diversity, the degree to which one agent tends to be correct when another is uncertain, and yields an upper bound on the gain available from choosing between the agents' answers on the basis of confidence, which we term Potential Conversation Synergy. Computed before any discussion, this metric was low in both settings where discussion did not help and high in both where it did: When knowledge-certainty diversity is low (as in LLM-only groups), discussion doesn't improve performance; when it is present (as in human or human-LLM groups), open-ended dialogue reliably lifts accuracy. These findings suggest a diagnostic for AI collaboration: knowledge-certainty diversity can be measured from pre-discussion responses and provides a configuration-level indicator of conversational gains across the settings studied.