4 citations · 7 across the 14 of their papers we have counts for
18 papers
Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity
Daniel Lee, Harsh Sharma, Eunkyu Park +5
MLLM-as-a-Judge is conventionally validated by agreement with human annotations, but this metric is undefined when the human pool is culturally heterogeneous. We introduce VOIR DIR…
The Role of Ambiguity in Error Prediction via Uncertainty Quantification
Ieva Raminta Staliūnaitė, James Bishop, Andreas Vlachos
The task of Error Prediction, namely predicting whether a model output is correct, is commonly tackled with Uncertainty Quantification (UQ). However, while uncertainty metrics capt…
Demystifying Multi-Agent Debate: The Role of Confidence and Diversity
Xiaochen Zhu, Caiqi Zhang, Yizhou Chi +3
Multi-agent debate (MAD) is widely used to improve large language model (LLM) performance through test-time scaling, yet recent work shows that vanilla MAD often underperforms simp…
Uncertainty Quantification for Evaluating Machine Translation Bias
Ieva Raminta Staliūnaitė, Julius Cheng, Andreas Vlachos
The predictive uncertainty of machine translation (MT) models is typically used as a quality estimation proxy. In this work, we posit that apart from confidently translating when a…
Causal Estimation of Tokenisation Bias
Pietro Lesci, Clara Meister, Thomas Hofmann +2
Modern language models are typically trained over subword sequences, but ultimately define probabilities over character-strings. Ideally, the choice of the tokeniser -- which maps…
LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generations
Caiqi Zhang, Xiaochen Zhu, Chengzu Li +2
Hallucination remains a major challenge for the safe and trustworthy deployment of large language models (LLMs) in factual content generation. Prior work has explored confidence es…
Collaborative Evaluation of Deepfake Text with Deliberation-Enhancing Dialogue Systems
Jooyoung Lee, Xiaochen Zhu, Georgi Karadzhov +3
The proliferation of generative models has presented significant challenges in distinguishing authentic human-authored content from deepfake content. Collaborative human efforts, a…
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
Liang Chen, Zekun Wang, Shuhuai Ren +24
Building on the foundations of language modeling in natural language processing, Next Token Prediction (NTP) has evolved into a versatile training objective for machine learning ta…
A Bayesian Optimization Approach to Machine Translation Reranking
Julius Cheng, Maike Züfle, Vilém Zouhar +1
Reranking a list of candidates from a machine translation system with an external scoring model and returning the highest-scoring candidate remains a simple and effective method fo…
Conformity in Large Language Models
Xiaochen Zhu, Caiqi Zhang, Tom Stafford +2
The conformity effect describes the tendency of individuals to align their responses with the majority. Studying this bias in large language models (LLMs) is crucial, as LLMs are i…
Causal Estimation of Memorisation Profiles
Pietro Lesci, Clara Meister, Thomas Hofmann +2
Understanding memorisation in language models has practical and societal implications, e.g., studying models' training dynamics or preventing copyright infringements. Prior work de…
An LLM Feature-based Framework for Dialogue Constructiveness Assessment
Lexin Zhou, Youmna Farag, Andreas Vlachos
Research on dialogue constructiveness assessment focuses on (i) analysing conversational factors that influence individuals to take specific actions, win debates, change their pers…
AnchorAL: Computationally Efficient Active Learning for Large and Imbalanced Datasets
Pietro Lesci, Andreas Vlachos
Active learning for imbalanced classification tasks is challenging as the minority classes naturally occur rarely. Gathering a large pool of unlabelled data is thus essential to ca…
The effect of diversity on group decision-making
Georgi Karadzhov, Andreas Vlachos, Tom Stafford
We explore different aspects of cognitive diversity and its effect on the success of group deliberation. To evaluate this, we use 500 dialogues from small, online groups discussing…
Do We Need Language-Specific Fact-Checking Models? The Case of Chinese
Caiqi Zhang, Zhijiang Guo, Andreas Vlachos
This paper investigates the potential benefits of language-specific fact-checking models, focusing on the case of Chinese. We first demonstrate the limitations of translation-based…
Faster Minimum Bayes Risk Decoding with Confidence-based Pruning
Julius Cheng, Andreas Vlachos
Minimum Bayes risk (MBR) decoding outputs the hypothesis with the highest expected utility over the model distribution for some utility function. It has been shown to improve accur…
Automated Fact-Checking in Dialogue: Are Specialized Models Needed?
Eric Chamoun, Marzieh Saeidi, Andreas Vlachos
Prior research has shown that typical fact-checking models for stand-alone claims struggle with claims made in dialogues. As a solution, fine-tuning these models on labelled dialog…
QA-NatVer: Question Answering for Natural Logic-based Fact Verification
Rami Aly, Marek Strong, Andreas Vlachos
Fact verification systems assess a claim's veracity based on evidence. An important consideration in designing them is faithfulness, i.e. generating explanations that accurately re…