7 papers
Investigating the Interplay between Contextual and Parametric Chain-of-Thought Faithfulness under Optimization
Jingyi Sun, Qianli Wang, Pepa Atanasova +2
Chain-of-Thought (CoT) faithfulness, i.e., whether CoTs genuinely reflect large language models' (LLM) underlying behavior, is typically evaluated with metrics under two disjoint p…
CUB: Benchmarking Context Utilisation Techniques for Language Models
Lovisa Hagström, Youna Kim, Haeun Yu +4
Incorporating external knowledge is crucial for knowledge-intensive tasks, such as question answering and fact checking. However, language models (LMs) may ignore relevant informat…
Evaluation Framework for Highlight Explanations of Context Utilisation in Language Models
Jingyi Sun, Pepa Atanasova, Sagnik Ray Choudhury +2
Context utilisation, the ability of Language Models (LMs) to incorporate relevant information from the provided context when generating responses, remains largely opaque to users,…
Multi-Step Knowledge Interaction Analysis via Rank-2 Subspace Disentanglement
Sekh Mainul Islam, Pepa Atanasova, Isabelle Augenstein
Natural Language Explanations (NLEs) describe how Large Language Models (LLMs) make decisions by drawing on external Context Knowledge (CK) and Parametric Knowledge (PK). Understan…
A Reality Check on Context Utilisation for Retrieval-Augmented Generation
Lovisa Hagström, Sara Vera MarjanoviÄ, Haeun Yu +5
Retrieval-augmented generation (RAG) helps address the limitations of parametric knowledge embedded within a language model (LM). In real world settings, retrieved information can…
Graph-Guided Textual Explanation Generation Framework
Shuzhou Yuan, Jingyi Sun, Ran Zhang +4
Natural language explanations (NLEs) are commonly used to provide plausible free-text explanations of a model's reasoning about its predictions. However, recent work has questioned…