Publications (44)
Interpretable Coreference Resolution Evaluation Using Explicit Semantics
Bruno Gatti, Giuliano Martinelli, Roberto Navigli
Coreference resolution is typically evaluated using aggregate statistical metrics such as CoNLL-F1, which measure structural overlap between predicted and gold clusters. While wide…
Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation
Luca Moroni, Giovanni Puccetti, Pere-Lluis Huguet Cabot +6
The number of pretrained Large Language Models (LLMs) is increasing steadily, though the majority are designed predominantly for the English language. While state-of-the-art LLMs c…
ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering
Francesco Maria Molfese, Luca Moroni, Ciro Porcaro +2
While Small Language Models (SLMs) have demonstrated promising performance on an increasingly wide array of commonsense reasoning benchmarks, current evaluation practices rely almo…
EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledge
Klim Zaporojets, Daniel Daza, Edoardo Barba +3
Knowledge Graphs (KGs) are structured knowledge repositories containing entities and relations between them. In this paper, we study the problem of automatically updating KGs over…
Exploring Non-Verbal Predicates in Semantic Role Labeling: Challenges and Opportunities
Riccardo Orlando, Simone Conia, Roberto Navigli
Although we have witnessed impressive progress in Semantic Role Labeling (SRL), most of the research in the area is carried out assuming that the majority of predicates are verbs.…
A Large-Scale Multilingual Disambiguation of Glosses
José Camacho Collados, Claudio Delli Bovi, Alessandro Raganato +1
Linking concepts and named entities to knowledge bases has become a crucial Natural Language Understanding task. In this respect, recent works have shown the key advantage of explo…
Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering
Francesco Maria Molfese, Luca Moroni, Luca Gioffré +3
One of the most widely used tasks for evaluating Large Language Models (LLMs) is Multiple-Choice Question Answering (MCQA). While open-ended question answering tasks are more chall…
BOOKCOREF: Coreference Resolution at Book Scale
Giuliano Martinelli, Tommaso Bonomo, Pere-LluÃs Huguet Cabot +1
Coreference Resolution systems are typically evaluated on benchmarks containing small- to medium-scale documents. When it comes to evaluating long texts, however, existing benchmar…
Cross-lingual AMR Aligner: Paying Attention to Cross-Attention
Abelardo Carlos MartÃnez Lorenzo, Pere-LluÃs Huguet Cabot, Roberto Navigli
This paper introduces a novel aligner for Abstract Meaning Representation (AMR) graphs that can scale cross-lingually, and is thus capable of aligning units and spans in sentences…
FuDoBa: Fusing Document and Knowledge Graph-based Representations with Bayesian Optimisation
Boshko Koloski, Senja Pollak, Roberto Navigli +1
Building on the success of Large Language Models (LLMs), LLM-based representations have dominated the document representation landscape, achieving great performance on the document…
Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-Oasis
Alessandro Scirè, Andrei Stefan Bejgu, Simone Tedeschi +3
After the introduction of Large Language Models (LLMs), there have been substantial improvements in the performance of Natural Language Generation (NLG) tasks, including Text Summa…
ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming
Simone Tedeschi, Felix Friedrich, Patrick Schramowski +4
When building Large Language Models (LLMs), it is paramount to bear safety in mind and protect them with guardrails. Indeed, LLMs should never generate content promoting or normali…
Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trends
Giuliano Martinelli, Edoardo Barba, Roberto Navigli
Large autoregressive generative models have emerged as the cornerstone for achieving the highest performance across several Natural Language Processing tasks. However, the urge to…
Knowledge Graphs
Aidan Hogan, Eva Blomqvist, Michael Cochez +15
In this paper we provide a comprehensive introduction to knowledge graphs, which have recently garnered significant attention from both industry and academia in scenarios that requ…
Code-Switching with Word Senses for Pretraining in Neural Machine Translation
Vivek Iyer, Edoardo Barba, Alexandra Birch +2
Lexical ambiguity is a significant and pervasive challenge in Neural Machine Translation (NMT), with many state-of-the-art (SOTA) NMT systems struggling to handle polysemous words…
FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction
Alessandro Scirè, Karim Ghonim, Roberto Navigli
Recent advancements in text summarization, particularly with the advent of Large Language Models (LLMs), have shown remarkable performance. However, a notable challenge persists as…
Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!
Stefano Perrella, Lorenzo Proietti, Alessandro Scirè +2
Annually, at the Conference of Machine Translation (WMT), the Metrics Shared Task organizers conduct the meta-evaluation of Machine Translation (MT) metrics, ranking them according…
Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress
Lorenzo Proietti, Stefano Perrella, Roberto Navigli
In Machine Translation (MT) evaluation, metric performance is assessed based on agreement with human judgments. In recent years, automatic metrics have demonstrated increasingly hi…
LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA
Tommaso Bonomo, Luca Gioffré, Roberto Navigli
Question Answering (QA) on narrative text poses a unique challenge to current systems, requiring a deep understanding of long, complex documents. However, the reliability of Narrat…
Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards
Raffaele Pisano, Roberto Navigli
Process Reward Models (PRMs) have emerged as a powerful tool for providing step-level feedback when evaluating the reasoning of Large Language Models (LLMs), which frequently produ…
Incorporating Graph Information in Transformer-based AMR Parsing
Pavlo Vasylenko, Pere-LluÃs Huguet Cabot, Abelardo Carlos MartÃnez Lorenzo +1
Abstract Meaning Representation (AMR) is a Semantic Parsing formalism that aims at providing a semantic graph abstraction representing a given text. Current approaches are based on…
Semantic Role Labeling Meets Definition Modeling: Using Natural Language to Describe Predicate-Argument Structures
Simone Conia, Edoardo Barba, Alessandro Scirè +1
One of the common traits of past and present approaches for Semantic Role Labeling (SRL) is that they rely upon discrete labels drawn from a predefined linguistic inventory to clas…
Do Large Language Models Understand Word Senses?
Domenico Meconi, Simone Stirpe, Federico Martelli +2
Understanding the meaning of words in context is a fundamental capability for Large Language Models (LLMs). Despite extensive evaluation efforts, the extent to which LLMs show evid…
Echoes from Alexandria: A Large Resource for Multilingual Book Summarization
Alessandro Scirè, Simone Conia, Simone Ciciliano +1
In recent years, research in text summarization has mainly focused on the news domain, where texts are typically short and have strong layout features. The task of full-book summar…
Huge Automatically Extracted Training Sets for Multilingual Word Sense Disambiguation
Tommaso Pasini, Francesco Maria Elia, Roberto Navigli
We release to the community six large-scale sense-annotated datasets in multiple language to pave the way for supervised multilingual Word Sense Disambiguation. Our datasets cover…
Estimating Machine Translation Difficulty
Lorenzo Proietti, Stefano Perrella, Vilém Zouhar +2
Machine translation quality has steadily improved over the years, achieving near-perfect translations in recent benchmarks. These high-quality outputs make it difficult to distingu…
LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies
Felix Friedrich, Simone Tedeschi, Patrick Schramowski +5
Building safe Large Language Models (LLMs) across multiple languages is essential in ensuring both safe access and linguistic diversity. To this end, we conduct a large-scale, comp…
EUREKA: EUphemism Recognition Enhanced through Knn-based methods and Augmentation
Sedrick Scott Keh, Rohit K. Bharadwaj, Emmy Liu +3
We introduce EUREKA, an ensemble-based approach for performing automatic euphemism detection. We (1) identify and correct potentially mislabelled rows in the dataset, (2) curate an…
Entity Disambiguation with Entity Definitions
Luigi Procopio, Simone Conia, Edoardo Barba +1
Local models have recently attained astounding performances in Entity Disambiguation (ED), with generative and extractive formulations being the most promising research directions.…
ZEBRA: Zero-Shot Example-Based Retrieval Augmentation for Commonsense Question Answering
Francesco Maria Molfese, Simone Conia, Riccardo Orlando +1
Current Large Language Models (LLMs) have shown strong reasoning capabilities in commonsense question answering benchmarks, but the process underlying their success remains largely…
AutoML-guided Fusion of Entity and LLM-based Representations for Document Classification
Boshko Koloski, Senja Pollak, Roberto Navigli +1
Large semantic knowledge bases are grounded in factual knowledge. However, recent approaches to dense text representations (i.e. embeddings) do not efficiently exploit these resour…
Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics
Stefano Perrella, Lorenzo Proietti, Pere-LluÃs Huguet Cabot +2
Machine Translation (MT) evaluation metrics assess translation quality automatically. Recently, researchers have employed MT metrics for various new use cases, such as data filteri…
Towards a Seamless Integration of Word Senses into Downstream NLP Applications
Mohammad Taher Pilehvar, Jose Camacho-Collados, Roberto Navigli +1
Lexical ambiguity can impede NLP systems from accurate understanding of semantics. Despite its potential benefits, the integration of sense-level information into NLP systems has r…
RED: a Filtered and Multilingual Relation Extraction Dataset
Pere-LluÃs Huguet Cabot, Simone Tedeschi, Axel-Cyrille Ngonga Ngomo +1
Relation Extraction (RE) is a task that identifies relationships between entities in a text, enabling the acquisition of relational facts and bridging the gap between natural langu…
What's the Meaning of Superhuman Performance in Today's NLU?
Simone Tedeschi, Johan Bos, Thierry Declerck +9
In the last five years, there has been a significant focus in Natural Language Processing (NLP) on developing larger Pretrained Language Models (PLMs) and introducing benchmarks su…
Word Sense Linking: Disambiguating Outside the Sandbox
Andrei Stefan Bejgu, Edoardo Barba, Luigi Procopio +2
Word Sense Disambiguation (WSD) is the task of associating a word in a given context with its most suitable meaning among a set of possible candidates. While the task has recently…
Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
Soyeon Caren Han, Feiqi Cao, Josiah Poon +1
This tutorial explores recent advancements in multimodal pretrained and large models, capable of integrating and processing diverse data forms such as text, images, audio, and vide…
Focusing on Context is NICE: Improving Overshadowed Entity Disambiguation
Vera Provatorova, Simone Tedeschi, Svitlana Vakulenko +2
Entity disambiguation (ED) is the task of mapping an ambiguous entity mention to the corresponding entry in a structured knowledge base. Previous research showed that entity oversh…
Embedding Words and Senses Together via Joint Knowledge-Enhanced Training
Massimiliano Mancini, Jose Camacho-Collados, Ignacio Iacobacci +1
Word embeddings are widely used in Natural Language Processing, mainly due to their success in capturing semantic information from massive corpora. However, their creation process…
The CQC Algorithm: Cycling in Graphs to Semantically Enrich and Enhance a Bilingual Dictionary
Tiziano Flati, Roberto Navigli
Bilingual machine-readable dictionaries are knowledge resources useful in many automatic tasks. However, compared to monolingual computational lexicons like WordNet, bilingual dict…
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
Taishi Nakamura, Mayank Mishra, Simone Tedeschi +42
Pretrained language models are an integral part of AI applications, but their high computational cost for training limits accessibility. Initiatives such as Bloom and StarCoder aim…
ReLiK: Retrieve and LinK, Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget
Riccardo Orlando, Pere-Lluis Huguet Cabot, Edoardo Barba +1
Entity Linking (EL) and Relation Extraction (RE) are fundamental tasks in Natural Language Processing, serving as critical components in a wide range of applications. In this paper…
AMRs Assemble! Learning to Ensemble with Autoregressive Models for AMR Parsing
Abelardo Carlos MartÃnez Lorenzo, Pere-LluÃs Huguet Cabot, Roberto Navigli
In this paper, we examine the current state-of-the-art in AMR parsing, which relies on ensemble strategies by merging multiple graph predictions. Our analysis reveals that the pres…
Semantic Representations of Word Senses and Concepts
José Camacho-Collados, Ignacio Iacobacci, Roberto Navigli +1
Representing the semantics of linguistic items in a machine-interpretable form has been a major goal of Natural Language Processing since its earliest days. Among the range of diff…