18 citations · 52 across the 20 of their papers we have counts for
15 papers · 1 filter
When Tools Get in the Way: The Effect of Unnecessary Tool Availability on LLM Answering
Saanvi Paturi, Arsen Kenzhebayev, Arham Sethi +3
Large language models (LLMs) are increasingly deployed with external tools that extend what they can do beyond their own knowledge. Tools help on tasks that need external informati…
Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models
Mahdiyeh Farajidizaji, Vatsal Raina
Instruction tuning is meant to make language models follow user requests, yet it is unclear whether small models comply when an instruction conflicts with their usual task behavior…
Probing the Limits of Stylistic Alignment in Vision-Language Models
Asma Farajidizaji, Akash Gupta, Vatsal Raina
Vision-language models are increasingly used to generate image captions in specific styles, such as humor or romantic. However, these transformer-based models often struggle with t…
Finetuning LLMs for Comparative Assessment Tasks
Vatsal Raina, Adian Liusie, Mark Gales
Automated assessment in natural language generation is a challenging task. Instruction-tuned large language models (LLMs) have shown promise in reference-free evaluation, particula…
Question-Based Retrieval using Atomic Units for Enterprise RAG
Vatsal Raina, Mark Gales
Enterprise retrieval augmented generation (RAG) offers a highly flexible framework for combining powerful large language models (LLMs) with internal, possibly temporally changing,…
Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisons
Adian Liusie, Vatsal Raina, Yassir Fathullah +1
LLM-as-a-judge approaches are a practical and effective way of assessing a range of text tasks. However, when using pairwise comparisons to rank a set of candidates, the computatio…
Question Difficulty Ranking for Multiple-Choice Reading Comprehension
Vatsal Raina, Mark Gales
Multiple-choice (MC) tests are an efficient method to assess English learners. It is useful for test creators to rank candidate MC questions by difficulty during exam curation. Typ…
An Information-Theoretic Approach to Analyze NLP Classification Tasks
Luran Wang, Mark Gales, Vatsal Raina
Understanding the importance of the inputs on the output is useful across many tasks. This work provides an information-theoretic framework to analyse the influence of inputs for t…
Assessing Distractors in Multiple-Choice Tests
Vatsal Raina, Adian Liusie, Mark Gales
Multiple-choice tests are a common approach for assessing candidates' comprehension skills. Standard multiple-choice reading comprehension exams require candidates to select the co…
Is it Possible to Modify Text to a Target Readability Level? An Initial Investigation Using Zero-Shot Large Language Models
Asma Farajidizaji, Vatsal Raina, Mark Gales
Text simplification is a common task where the text is adapted to make it easier to understand. Similarly, text elaboration can make a passage more sophisticated, offering a method…
Analyzing Multiple-Choice Reading and Listening Comprehension Tests
Vatsal Raina, Adian Liusie, Mark Gales
Multiple-choice reading and listening comprehension tests are an important part of language assessment. Content creators for standard educational tests need to carefully curate que…
CUED at ProbSum 2023: Hierarchical Ensemble of Summarization Models
Potsawee Manakul, Yassir Fathullah, Adian Liusie +3
In this paper, we consider the challenge of summarizing patients' medical progress notes in a limited data setting. For the Problem List Summarization (shared task 1A) at the BioNL…
Analysis of the Cambridge Multiple-Choice Questions Reading Dataset with a Focus on Candidate Response Distribution
Adian Liusie, Vatsal Raina, Andrew Mullooly +2
Multiple choice exams are widely used to assess candidates across a diverse range of domains and tasks. To moderate question quality, newly proposed questions often pass through pr…
World Knowledge in Multiple Choice Reading Comprehension
Adian Liusie, Vatsal Raina, Mark Gales
Recently it has been shown that without any access to the contextual passage, multiple choice reading comprehension (MCRC) systems are able to answer questions significantly better…
Multiple-Choice Question Generation: Towards an Automated Assessment Framework
Vatsal Raina, Mark Gales
Automated question generation is an important approach to enable personalisation of English comprehension assessment. Recently, transformer-based pretrained language models have de…