7 papers · 1 filter
What Does Loss Optimization Actually Teach, If Anything? Knowledge Dynamics in Continual Pre-training of LLMs
Seyed Mahed Mousavi, Simone Alghisi, Giuseppe Riccardi
Continual Pre-Training (CPT) is widely used for acquiring and updating factual knowledge in LLMs. This practice treats loss as a proxy for knowledge learning, while offering no gro…
Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It
Seyed Mahed Mousavi, Edoardo Cecchinato, Lucia Hornikova +1
We conduct a systematic audit of three widely used reasoning benchmarks, SocialIQa, FauxPas-EAI, and ToMi, and uncover pervasive flaws in both benchmark items and evaluation method…
LLMs as Repositories of Factual Knowledge: Limitations and Solutions
Seyed Mahed Mousavi, Simone Alghisi, Giuseppe Riccardi
LLMs' sources of knowledge are data snapshots containing factual information about entities collected at different timestamps and from different media types (e.g. wikis, social med…
Will LLMs Replace the Encoder-Only Models in Temporal Relation Classification?
Gabriel Roccabruna, Massimo Rizzoli, Giuseppe Riccardi
The automatic detection of temporal relations among events has been mainly investigated with encoder-only models such as RoBERTa. Large Language Models (LLM) have recently shown pr…
Should We Fine-Tune or RAG? Evaluating Different Techniques to Adapt LLMs for Dialogue
Simone Alghisi, Massimo Rizzoli, Gabriel Roccabruna +2
We study the limitations of Large Language Models (LLMs) for the task of response generation in human-machine dialogue. Several techniques have been proposed in the literature for…
DyKnow: Dynamically Verifying Time-Sensitive Factual Knowledge in LLMs
Seyed Mahed Mousavi, Simone Alghisi, Giuseppe Riccardi
LLMs acquire knowledge from massive data snapshots collected at different timestamps. Their knowledge is then commonly evaluated using static benchmarks. However, factual knowledge…