On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decode

Valeria Ruscio, Keiran Thompson

A language model can give the wrong answer even when the correct answer is decodable from its intermediate states. To study this gap between decodability and selection, we distingu…