9 citations · 29 across the 12 of their papers we have counts for
14 papers · 1 filter
Evaluating the Meta- and Object-Level Reasoning of Large Language Models for Question Answering
Nick Ferguson, Liane Guillou, Alan Bundy +1
Large Language Models (LLMs) excel in natural language tasks but still face challenges in Question Answering (QA) tasks requiring complex, multi-step reasoning. We outline the type…
Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets
Nikita Moghe, Arnisa Fazla, Chantal Amrhein +5
Recent machine translation (MT) metrics calibrate their effectiveness by correlating with human judgement but without any insights about their behaviour across different error type…
ACES: Translation Accuracy Challenge Sets at WMT 2023
Chantal Amrhein, Nikita Moghe, Liane Guillou
We benchmark the performance of segmentlevel metrics submitted to WMT 2023 using the ACES Challenge Set (Amrhein et al., 2022). The challenge set consists of 36K examples represent…
MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialogue
Nikita Moghe, Evgeniia Razumovskaia, Liane Guillou +3
Task-oriented dialogue (TOD) systems have been widely deployed in many industries as they deliver more efficient customer support. These systems are typically constructed for a sin…
ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics
Chantal Amrhein, Nikita Moghe, Liane Guillou
As machine translation (MT) metrics improve their correlation with human judgement every year, it is crucial to understand the limitations of such metrics at the segment level. Spe…
Investigating the use of Paraphrase Generation for Question Reformulation in the FRANK QA system
Nick Ferguson, Liane Guillou, Kwabena Nuamah +1
We present a study into the ability of paraphrase generation methods to increase the variety of natural language questions that the FRANK Question Answering system can answer. We f…
Cross-lingual Inference with A Chinese Entailment Graph
Tianyi Li, Sabine Weber, Mohammad Javad Hosseini +2
Predicate entailment detection is a crucial task for question-answering from text, where previous work has explored unsupervised learning of entailment graphs from typed open relat…
Blindness to Modality Helps Entailment Graph Mining
Liane Guillou, Sander Bijl de Vroe, Mark Johnson +1
Understanding linguistic modality is widely seen as important for downstream tasks such as Question Answering and Knowledge Graph Population. Entailment Graph learning might also b…
Incorporating Temporal Information in Entailment Graph Mining
Liane Guillou, Sander Bijl de Vroe, Mohammad Javad Hosseini +2
We present a novel method for injecting temporality into entailment graphs to address the problem of spurious entailments, which may arise from similar but temporally distinct even…
Modality and Negation in Event Extraction
Sander Bijl de Vroe, Liane Guillou, Miloš Stanojević +2
Language provides speakers with a rich system of modality for expressing thoughts about events, without being committed to their actual occurrence. Modality is commonly used in the…
Multivalent Entailment Graphs for Question Answering
Nick McKenna, Liane Guillou, Mohammad Javad Hosseini +3
Drawing inferences between open-domain natural language predicates is a necessity for true language understanding. There has been much progress in unsupervised learning of entailme…
Findings of the 2016 WMT Shared Task on Cross-lingual Pronoun Prediction
Liane Guillou, Christian Hardmeier, Preslav Nakov +6
We describe the design, the evaluation setup, and the results of the 2016 WMT shared task on cross-lingual pronoun prediction. This is a classification task in which participants a…
Pronoun Translation in English-French Machine Translation: An Analysis of Error Types
Christian Hardmeier, Liane Guillou
Pronouns are a long-standing challenge in machine translation. We present a study of the performance of a range of rule-based, statistical and neural MT systems on pronoun translat…
Automatic Reference-Based Evaluation of Pronoun Translation Misses the Point
Liane Guillou, Christian Hardmeier
We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translat…