3 citations · 14 across the 29 of their papers we have counts for
30 papers
LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA
Tommaso Bonomo, Luca Gioffré, Roberto Navigli
Question Answering (QA) on narrative text poses a unique challenge to current systems, requiring a deep understanding of long, complex documents. However, the reliability of Narrat…
ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering
Francesco Maria Molfese, Luca Moroni, Ciro Porcaro +2
While Small Language Models (SLMs) have demonstrated promising performance on an increasingly wide array of commonsense reasoning benchmarks, current evaluation practices rely almo…
Do Large Language Models Understand Word Senses?
Domenico Meconi, Simone Stirpe, Federico Martelli +2
Understanding the meaning of words in context is a fundamental capability for Large Language Models (LLMs). Despite extensive evaluation efforts, the extent to which LLMs show evid…
Estimating Machine Translation Difficulty
Lorenzo Proietti, Stefano Perrella, Vilém Zouhar +2
Machine translation quality has steadily improved over the years, achieving near-perfect translations in recent benchmarks. These high-quality outputs make it difficult to distingu…
BOOKCOREF: Coreference Resolution at Book Scale
Giuliano Martinelli, Tommaso Bonomo, Pere-Lluís Huguet Cabot +1
Coreference Resolution systems are typically evaluated on benchmarks containing small- to medium-scale documents. When it comes to evaluating long texts, however, existing benchmar…
FuDoBa: Fusing Document and Knowledge Graph-based Representations with Bayesian Optimisation
Boshko Koloski, Senja Pollak, Roberto Navigli +1
Building on the success of Large Language Models (LLMs), LLM-based representations have dominated the document representation landscape, achieving great performance on the document…
EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledge
Klim Zaporojets, Daniel Daza, Edoardo Barba +3
Knowledge Graphs (KGs) are structured knowledge repositories containing entities and relations between them. In this paper, we study the problem of automatically updating KGs over…
Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress
Lorenzo Proietti, Stefano Perrella, Roberto Navigli
In Machine Translation (MT) evaluation, metric performance is assessed based on agreement with human judgments. In recent years, automatic metrics have demonstrated increasingly hi…
Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation
Luca Moroni, Giovanni Puccetti, Pere-Lluis Huguet Cabot +6
The number of pretrained Large Language Models (LLMs) is increasing steadily, though the majority are designed predominantly for the English language. While state-of-the-art LLMs c…
Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering
Francesco Maria Molfese, Luca Moroni, Luca Gioffré +3
One of the most widely used tasks for evaluating Large Language Models (LLMs) is Multiple-Choice Question Answering (MCQA). While open-ended question answering tasks are more chall…
Word Sense Linking: Disambiguating Outside the Sandbox
Andrei Stefan Bejgu, Edoardo Barba, Luigi Procopio +2
Word Sense Disambiguation (WSD) is the task of associating a word in a given context with its most suitable meaning among a set of possible candidates. While the task has recently…
LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies
Felix Friedrich, Simone Tedeschi, Patrick Schramowski +5
Building safe Large Language Models (LLMs) across multiple languages is essential in ensuring both safe access and linguistic diversity. To this end, we conduct a large-scale, comp…
Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-Oasis
Alessandro Scirè, Andrei Stefan Bejgu, Simone Tedeschi +3
After the introduction of Large Language Models (LLMs), there have been substantial improvements in the performance of Natural Language Generation (NLG) tasks, including Text Summa…
Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics
Stefano Perrella, Lorenzo Proietti, Pere-Lluís Huguet Cabot +2
Machine Translation (MT) evaluation metrics assess translation quality automatically. Recently, researchers have employed MT metrics for various new use cases, such as data filteri…
ZEBRA: Zero-Shot Example-Based Retrieval Augmentation for Commonsense Question Answering
Francesco Maria Molfese, Simone Conia, Riccardo Orlando +1
Current Large Language Models (LLMs) have shown strong reasoning capabilities in commonsense question answering benchmarks, but the process underlying their success remains largely…
AutoML-guided Fusion of Entity and LLM-based Representations for Document Classification
Boshko Koloski, Senja Pollak, Roberto Navigli +1
Large semantic knowledge bases are grounded in factual knowledge. However, recent approaches to dense text representations (i.e. embeddings) do not efficiently exploit these resour…
Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!
Stefano Perrella, Lorenzo Proietti, Alessandro Scirè +2
Annually, at the Conference of Machine Translation (WMT), the Metrics Shared Task organizers conduct the meta-evaluation of Machine Translation (MT) metrics, ranking them according…
ReLiK: Retrieve and LinK, Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget
Riccardo Orlando, Pere-Lluis Huguet Cabot, Edoardo Barba +1
Entity Linking (EL) and Relation Extraction (RE) are fundamental tasks in Natural Language Processing, serving as critical components in a wide range of applications. In this paper…
Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trends
Giuliano Martinelli, Edoardo Barba, Roberto Navigli
Large autoregressive generative models have emerged as the cornerstone for achieving the highest performance across several Natural Language Processing tasks. However, the urge to…
ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming
Simone Tedeschi, Felix Friedrich, Patrick Schramowski +4
When building Large Language Models (LLMs), it is paramount to bear safety in mind and protect them with guardrails. Indeed, LLMs should never generate content promoting or normali…
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
Taishi Nakamura, Mayank Mishra, Simone Tedeschi +42
Pretrained language models are an integral part of AI applications, but their high computational cost for training limits accessibility. Initiatives such as Bloom and StarCoder aim…
FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction
Alessandro Scirè, Karim Ghonim, Roberto Navigli
Recent advancements in text summarization, particularly with the advent of Large Language Models (LLMs), have shown remarkable performance. However, a notable challenge persists as…
Code-Switching with Word Senses for Pretraining in Neural Machine Translation
Vivek Iyer, Edoardo Barba, Alexandra Birch +2
Lexical ambiguity is a significant and pervasive challenge in Neural Machine Translation (NMT), with many state-of-the-art (SOTA) NMT systems struggling to handle polysemous words…
Exploring Non-Verbal Predicates in Semantic Role Labeling: Challenges and Opportunities
Riccardo Orlando, Simone Conia, Roberto Navigli
Although we have witnessed impressive progress in Semantic Role Labeling (SRL), most of the research in the area is carried out assuming that the majority of predicates are verbs.…