4 papers
What Does Loss Optimization Actually Teach, If Anything? Knowledge Dynamics in Continual Pre-training of LLMs
Seyed Mahed Mousavi, Simone Alghisi, Giuseppe Riccardi
Continual Pre-Training (CPT) is widely used for acquiring and updating factual knowledge in LLMs. This practice treats loss as a proxy for knowledge learning, while offering no gro…
[De|Re]constructing VLMs' Reasoning in Counting
Simone Alghisi, Gabriel Roccabruna, Massimo Rizzoli +2
Vision-Language Models (VLMs) have recently gained attention due to their competitive performance on multiple downstream tasks, achieved by following user-input instructions. Howev…
Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It
Seyed Mahed Mousavi, Edoardo Cecchinato, Lucia Hornikova +1
We conduct a systematic audit of three widely used reasoning benchmarks, SocialIQa, FauxPas-EAI, and ToMi, and uncover pervasive flaws in both benchmark items and evaluation method…
CIVET: Systematic Evaluation of Understanding in VLMs
Massimo Rizzoli, Simone Alghisi, Olha Khomyn +3
While Vision-Language Models (VLMs) have achieved competitive performance in various tasks, their comprehension of the underlying structure and semantics of a scene remains underst…