18 citations · 52 across the 20 of their papers we have counts for
23 papers
Fabrication After Tool Failure: Tool-Augmented Agents Assert Values Their Tools Did Not Return
Arham Sethi, Arsen Kenzhebayev, Saanvi Paturi +3
Tool-augmented language models are evaluated on whether they reach the right answer, not on whether they report honestly when a tool fails to supply one. We isolate this post-failu…
When Tools Get in the Way: The Effect of Unnecessary Tool Availability on LLM Answering
Saanvi Paturi, Arsen Kenzhebayev, Arham Sethi +3
Large language models (LLMs) are increasingly deployed with external tools that extend what they can do beyond their own knowledge. Tools help on tasks that need external informati…
Knowing When Not to Answer: Pseudo-Ensembles for Abstention in Music Audio-Language Models
Aanya Maheshwari, Vatsal Raina
Music audio-language models are evaluated almost entirely by accuracy on multiple-choice questions. This protocol forces the model to commit to an option, so a lucky guess looks th…
Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models
Mahdiyeh Farajidizaji, Vatsal Raina
Instruction tuning is meant to make language models follow user requests, yet it is unclear whether small models comply when an instruction conflicts with their usual task behavior…
Embedding the Teacher: Distilling vLLM Preferences for Scalable Image Retrieval
Eric He, Akash Gupta, Adian Liusie +4
Text--image retrieval is necessary for applications such as product recommendation. Embedding-based approaches like CLIP enable efficient large-scale retrieval via vector similarit…
Probing the Limits of Stylistic Alignment in Vision-Language Models
Asma Farajidizaji, Akash Gupta, Vatsal Raina
Vision-language models are increasingly used to generate image captions in specific styles, such as humor or romantic. However, these transformer-based models often struggle with t…
ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models
Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma +31
Large Multimodal Models (LMMs) exhibit shortfalls when interpreting images and, by some measures, have poorer spatial cognition than young children or animals. Despite this, they a…
Finetuning LLMs for Comparative Assessment Tasks
Vatsal Raina, Adian Liusie, Mark Gales
Automated assessment in natural language generation is a challenging task. Instruction-tuned large language models (LLMs) have shown promise in reference-free evaluation, particula…
Question-Based Retrieval using Atomic Units for Enterprise RAG
Vatsal Raina, Mark Gales
Enterprise retrieval augmented generation (RAG) offers a highly flexible framework for combining powerful large language models (LLMs) with internal, possibly temporally changing,…
Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisons
Adian Liusie, Vatsal Raina, Yassir Fathullah +1
LLM-as-a-judge approaches are a practical and effective way of assessing a range of text tasks. However, when using pairwise comparisons to rank a set of candidates, the computatio…
Question Difficulty Ranking for Multiple-Choice Reading Comprehension
Vatsal Raina, Mark Gales
Multiple-choice (MC) tests are an efficient method to assess English learners. It is useful for test creators to rank candidate MC questions by difficulty during exam curation. Typ…
An Information-Theoretic Approach to Analyze NLP Classification Tasks
Luran Wang, Mark Gales, Vatsal Raina
Understanding the importance of the inputs on the output is useful across many tasks. This work provides an information-theoretic framework to analyse the influence of inputs for t…
Assessing Distractors in Multiple-Choice Tests
Vatsal Raina, Adian Liusie, Mark Gales
Multiple-choice tests are a common approach for assessing candidates' comprehension skills. Standard multiple-choice reading comprehension exams require candidates to select the co…
Structural-Based Uncertainty in Deep Learning Across Anatomical Scales: Analysis in White Matter Lesion Segmentation
Nataliia Molchanova, Vatsal Raina, Andrey Malinin +7
This paper explores uncertainty quantification (UQ) as an indicator of the trustworthiness of automated deep-learning (DL) tools in the context of white matter lesion (WML) segment…
Is it Possible to Modify Text to a Target Readability Level? An Initial Investigation Using Zero-Shot Large Language Models
Asma Farajidizaji, Vatsal Raina, Mark Gales
Text simplification is a common task where the text is adapted to make it easier to understand. Similarly, text elaboration can make a passage more sophisticated, offering a method…
Analyzing Multiple-Choice Reading and Listening Comprehension Tests
Vatsal Raina, Adian Liusie, Mark Gales
Multiple-choice reading and listening comprehension tests are an important part of language assessment. Content creators for standard educational tests need to carefully curate que…
CUED at ProbSum 2023: Hierarchical Ensemble of Summarization Models
Potsawee Manakul, Yassir Fathullah, Adian Liusie +3
In this paper, we consider the challenge of summarizing patients' medical progress notes in a limited data setting. For the Problem List Summarization (shared task 1A) at the BioNL…
Analysis of the Cambridge Multiple-Choice Questions Reading Dataset with a Focus on Candidate Response Distribution
Adian Liusie, Vatsal Raina, Andrew Mullooly +2
Multiple choice exams are widely used to assess candidates across a diverse range of domains and tasks. To moderate question quality, newly proposed questions often pass through pr…
Tackling Bias in the Dice Similarity Coefficient: Introducing nDSC for White Matter Lesion Segmentation
Vatsal Raina, Nataliia Molchanova, Mara Graziani +4
The development of automatic segmentation techniques for medical imaging tasks requires assessment metrics to fairly judge and rank such approaches on benchmarks. The Dice Similari…
Novel structural-scale uncertainty measures and error retention curves: application to multiple sclerosis
Nataliia Molchanova, Vatsal Raina, Andrey Malinin +6
This paper focuses on the uncertainty estimation for white matter lesions (WML) segmentation in magnetic resonance imaging (MRI). On one side, voxel-scale segmentation errors cause…
World Knowledge in Multiple Choice Reading Comprehension
Adian Liusie, Vatsal Raina, Mark Gales
Recently it has been shown that without any access to the contextual passage, multiple choice reading comprehension (MCRC) systems are able to answer questions significantly better…
Multiple-Choice Question Generation: Towards an Automated Assessment Framework
Vatsal Raina, Mark Gales
Automated question generation is an important approach to enable personalisation of English comprehension assessment. Recently, transformer-based pretrained language models have de…
Shifts 2.0: Extending The Dataset of Real Distributional Shifts
Andrey Malinin, Andreas Athanasopoulos, Muhamed Barakovic +15
Distributional shift, or the mismatch between training and deployment data, is a significant obstacle to the usage of machine learning in high-stakes industrial applications, such…