7 papers
LLJ Cards: Best practices for the Use of LLMs as Judges
Khaoula Chehbouni, Melina Medjdoub, Florian Carichon +2
In recent years, large language models (LLMs) have emerged as a popular alternative for evaluation. Often referred to as LLMs as judges (LLJs), these systems have been widely adopt…
Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation
Cesare Spinoso-Di Piano, Verna Dankers, Marius Mosbach +1
Human language is driven by unspoken beliefs and belief updates, making these critical to model for successful communication between large language models (LLMs) and their users. I…
Can Vision Language Models Be Adaptive in Mathematics Education? A Learner Model-based Rubric Study
Jie Gao, Yongan Yu, Junzhu Su +3
Adaptive learning refers to educational technologies that track learners' learning progress and adapt the instructional process based on individual learners' learning performance.…
Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
Khaoula Chehbouni, Mohammed Haddou, Jackie Chi Kit Cheung +1
Evaluating natural language generation (NLG) systems remains a core challenge of natural language processing (NLP), further complicated by the rise of large language models (LLMs)…
Does This Summary Answer My Question? Modeling Query-Focused Summary Readers with Rational Speech Acts
Cesare Spinoso-Di Piano, Jackie Chi Kit Cheung
Query-focused summarization (QFS) is the task of generating a summary in response to a user-written query. Despite its user-oriented nature, there has been limited work in QFS in e…
CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling
Yu Bai, Xiyuan Zou, Heyan Huang +4
Long sequence modeling has gained broad interest as large language models (LLMs) continue to advance. Recent research has identified that a large portion of hidden states within th…
Identifying and Analyzing Performance-Critical Tokens in Large Language Models
Yu Bai, Heyan Huang, Cesare Spinoso-Di Piano +4
In-context learning (ICL) has emerged as an effective solution for few-shot learning with large language models (LLMs). However, how LLMs leverage demonstrations to specify a task…