Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses

Xenia Ohmer, Elia Bruni, Dieuwke Hupkes

At the staggering pace with which the capabilities of large language models (LLMs) are increasing, creating future-proof evaluation sets to assess their understanding becomes more…