5 papers
To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning
Nevena LaziÄ, Liam Fowl, András György +1
We investigate the ability of decoder-only transformer models to perform abstract symbolic reasoning; specifically solving propositional logic reasoning problems given in-context.…
Frontier LLMs Still Struggle with Simple Reasoning Tasks
Alan Malek, Jiawei Ge, Nevena Lazic +3
While state-of-the-art large language models (LLMs) demonstrate advanced reasoning capabilities-achieving remarkable performance on challenging competitive math and coding benchmar…
Beyond Statistical Learning: Exact Learning Is Essential for General Intelligence
András György, Tor Lattimore, Nevena LaziÄ +1
Sound deductive reasoning -- the ability to derive new knowledge from existing facts and rules -- is an indisputably desirable aspect of general intelligence. Despite the major adv…
To Believe or Not to Believe Your LLM
Yasin Abbasi Yadkori, Ilja Kuzborskij, András György +1
We explore uncertainty quantification in large language models (LLMs), with the goal to identify when uncertainty in responses given a query is large. We simultaneously consider bo…
Mitigating LLM Hallucinations via Conformal Abstention
Yasin Abbasi Yadkori, Ilja Kuzborskij, David Stutz +9
We develop a principled procedure for determining when a large language model (LLM) should abstain from responding (e.g., by saying "I don't know") in a general domain, instead of…