45 citations · 302 across the 119 of their papers we have counts for
35 papers · 1 filter
Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation
Tianyu Liu, Jirui Qi, Paul He +3
Recent work suggests that large language models enhanced with retrieval-augmented generation are easily influenced by the order, in which the retrieved documents are presented to t…
SIKeD: Self-guided Iterative Knowledge Distillation for mathematical reasoning
Shivam Adarsh, Kumar Shridhar, Caglar Gulcehre +2
Large Language Models (LLMs) can transfer their reasoning skills to smaller models by teaching them to generate the intermediate reasoning process required to solve multistep reaso…
SMART: Self-learning Meta-strategy Agent for Reasoning Tasks
Rongxing Liu, Kumar Shridhar, Manish Prajapat +2
Tasks requiring deductive reasoning, especially those involving multiple steps, often demand adaptive strategies such as intermediate generation of rationales or programs, as no si…
Efficiently Computing Susceptibility to Context in Language Models
Tianyu Liu, Kevin Du, Mrinmaya Sachan +1
One strength of modern language models is their ability to incorporate information from a user-input context when answering queries. However, they are not equally sensitive to the…
LLM-based Cognitive Models of Students with Misconceptions
Shashank Sonkar, Xinghe Chen, Naiming Liu +2
Accurately modeling student cognition is crucial for developing effective AI-driven educational technologies. A key challenge is creating realistic student models that satisfy two…
MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofs
Andreas Opedal, Haruki Shirakami, Bernhard Schölkopf +2
Large language models (LLMs) can solve arithmetic word problems with high accuracy, but little is known about how well they generalize to more complex problems. This is difficult t…
Towards the Pedagogical Steering of Large Language Models for Tutoring: A Case Study with Modeling Productive Failure
Romain Puech, Jakub Macina, Julia Chatain +2
One-to-one tutoring is one of the most efficient methods of teaching. With the growing popularity of Large Language Models (LLMs), there have been efforts to create LLM based conve…
Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracing
Yilmazcan Ozyurt, Stefan Feuerriegel, Mrinmaya Sachan
Knowledge tracing (KT) is a popular approach for modeling students' learning progress over time, which can enable more personalized and adaptive learning. However, existing KT appr…
Do Vision-Language Models Really Understand Visual Language?
Yifan Hou, Buse Giledereli, Yilei Tu +1
Visual language is a system of communication that conveys information through symbols, shapes, and spatial arrangements. Diagrams are a typical example of a visual language depicti…
Retro-li: Small-Scale Retrieval Augmented Generation Supporting Noisy Similarity Searches and Domain Shift Generalization
Gentiana Rashiti, Geethan Karunaratne, Mrinmaya Sachan +2
The retrieval augmented generation (RAG) system such as Retro has been shown to improve language modeling capabilities and reduce toxicity and hallucinations by retrieving from a d…
GPT-4 as a Homework Tutor can Improve Student Engagement and Learning Outcomes
Alessandro Vanzo, Sankalan Pal Chowdhury, Mrinmaya Sachan
This work contributes to the scarce empirical literature on LLM-based interactive homework in real-world educational settings and offers a practical, scalable solution for improvin…
AI-assisted Automated Short Answer Grading of Handwritten University Level Mathematics Exams
Tianyi Liu, Julia Chatain, Laura Kobel-Keller +3
Effective and timely feedback in educational assessments is essential but labor-intensive, especially for complex tasks. Recent developments in automated feedback systems, ranging…
How to Engage Your Readers? Generating Guiding Questions to Promote Active Reading
Peng Cui, Vilém Zouhar, Xiaoyu Zhang +1
Using questions in written text is an effective strategy to enhance readability. However, what makes an active reading question good, what the linguistic role of these questions is…
Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors
Nico Daheim, Jakub Macina, Manu Kapur +2
Large language models (LLMs) present an opportunity to scale high-quality personalized education to all. A promising approach towards this means is to build dialog tutoring models…
Towards Aligning Language Models with Textual Feedback
Saüc Abadal Lloret, Shehzaad Dhuliawala, Keerthiram Murugesan +1
We present ALT (ALignment with Textual feedback), an approach that aligns language models with user preferences expressed in text. We argue that text offers greater expressiveness,…
Language Model Alignment in Multilingual Trolley Problems
Zhijing Jin, Max Kleiman-Weiner, Giorgio Piatti +9
We evaluate the moral alignment of LLMs with human preferences in multilingual trolley problems. Building on the Moral Machine experiment, which captures over 40 million human judg…
Confidence Regulation Neurons in Language Models
Alessandro Stolfo, Ben Wu, Wes Gurnee +4
Despite their widespread use, the mechanisms by which large language models (LLMs) represent and regulate uncertainty in next-token predictions remain largely unexplored. This stud…
DIRAS: Efficient LLM Annotation of Document Relevance in Retrieval Augmented Generation
Jingwei Ni, Tobias Schimanski, Meihong Lin +3
Retrieval Augmented Generation (RAG) is widely employed to ground responses to queries on domain-specific documents. But do RAG implementations leave out important information when…
AI-Assisted Human Evaluation of Machine Translation
Vilém Zouhar, Tom Kocmi, Mrinmaya Sachan
Annually, research teams spend large amounts of money to evaluate the quality of machine translation systems (WMT, inter alia). This is expensive because it requires a lot of exper…
Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation
Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis +5
High-quality Machine Translation (MT) evaluation relies heavily on human judgments. Comprehensive error classification methods, such as Multidimensional Quality Metrics (MQM), are…
What Do Language Models Learn in Context? The Structured Task Hypothesis
Jiaoda Li, Yifan Hou, Mrinmaya Sachan +1
Large language models (LLMs) exhibit an intriguing ability to learn a novel task from in-context examples presented in a demonstration, termed in-context learning (ICL). Understand…
On Affine Homotopy between Language Encoders
Robin SM Chan, Reda Boumasmoud, Anej Svete +8
Pre-trained language encoders -- functions that represent text as vectors -- are an integral component of many NLP tasks. We tackle a natural question in language encoder analysis:…
A Transformer with Stack Attention
Jiaoda Li, Jennifer C. White, Mrinmaya Sachan +1
Natural languages are believed to be (mildly) context-sensitive. Despite underpinning remarkably capable large language models, transformers are unable to model many context-free l…
Quriosity: Analyzing Human Questioning Behavior and Causal Inquiry through Curiosity-Driven Queries
Roberto Ceraolo, Dmitrii Kharlapenko, Ahmad Khan +6
Recent progress in Large Language Model (LLM) technology has changed our role in interacting with these models. Instead of primarily testing these models with questions we already…