32 citations · 85 across the 50 of their papers we have counts for
53 papers
SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
Pierre Beckmann, Marco Valentino, Andre Freitas
Three paradigmatic forms of inference recur across scientific reasoning: deduction, induction, and causal abduction. Reliably evaluating LLMs on these in scientific settings is cur…
Neurosymbolic Clinical Trial Matching via LLM-Driven Abduction and Logical Verification
Baiyang Qu, Leonardo Ranaldi, Xi Wang +1
Large Language Models (LLMs) offer a promising path to automate Clinical Trial Matching (CTM), but still struggle with the deterministic verification required for complex eligibili…
Is Inference Mediated by Distinct Semantic Structures in LLMs? A Mechanistic Interpretation
Nura Aljaafari, Marco Valentino, André Freitas
Predicting a label correctly does not necessarily require representing the operation that produces it. Transformer representations are known to carry label-level information, but w…
Compliance vs. Sensibility: On the Reasoning Controllability in Large Language Models
Xingwei Tan, Marco Valentino, Mahmud Elahi Akhter +3
Large Language Models (LLMs) acquire reasoning capabilities through shared inference patterns in pre-training data, which are further elicited via Chain-of-Thought (CoT). However,…
Decompose-and-Formalise: Recursively Verifiable Natural Language Inference
Xin Quan, Marco Valentino, Louise A. Dennis +1
Recent work has shown that integrating large language models (LLMs) with theorem provers (TPs) in neuro-symbolic pipelines helps with entailment verification and proof-guided refin…
Inferring Latent Intentions: Attributional Natural Language Inference in LLM Agents
Xin Quan, Jiafeng Xiong, Marco Valentino +1
Attributional inference, the ability to predict latent intentions behind observed actions, is a critical yet underexplored capability for large language models (LLMs) operating in…
Monotonic Reference-Free Refinement for Autoformalization
Lan Zhang, Marco Valentino, André Freitas
While statement autoformalization has advanced rapidly, full-theorem autoformalization remains largely unexplored. Existing iterative refinement methods in statement autoformalizat…
MASA: LLM-Driven Multi-Agent Systems for Autoformalization
Lan Zhang, Marco Valentino, André Freitas
Autoformalization serves a crucial role in connecting natural language and formal reasoning. This paper presents MASA, a novel framework for building multi-agent systems for autofo…
Compartmentalised Agentic Reasoning for Clinical NLI
Maël Jullien, Lei Xu, Marco Valentino +1
Large language models can produce fluent judgments for clinical natural language inference, yet they frequently fail when the decision requires the correct inferential schema rathe…
The Knowledge-Reasoning Dissociation: Fundamental Limitations of LLMs in Clinical Natural Language Inference
Maël Jullien, Marco Valentino, André Freitas
Large language models are often assumed to acquire increasingly structured, generalizable internal representations simply by scaling data and parameters. We interrogate this assump…
Dissecting Clinical Reasoning in Language Models: A Comparative Study of Prompts and Model Adaptation Strategies
Mael Jullien, Marco Valentino, Leonardo Ranaldi +1
Recent works on large language models (LLMs) have demonstrated the impact of prompting strategies and fine-tuning techniques on their reasoning capabilities. Yet, their effectivene…
Beyond Gold Standards: Epistemic Ensemble of LLM Judges for Formal Mathematical Reasoning
Lan Zhang, Marco Valentino, Jordan Meadows +1
Statement autoformalization plays a crucial role in formal mathematical reasoning by enabling the automatic translation of natural language statements into formal languages. While…
Faithful and Robust LLM-Driven Theorem Proving for NLI Explanations
Xin Quan, Marco Valentino, Louise A. Dennis +1
Natural language explanations play a fundamental role in Natural Language Inference (NLI) by revealing how premises logically entail hypotheses. Recent work has shown that the inte…
Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation Steering
Marco Valentino, Geonhee Kim, Dhairya Dalal +2
Large language models (LLMs) exhibit reasoning biases, often conflating content plausibility with formal logical validity. This can lead to wrong inferences in critical domains, wh…
PEIRCE: Unifying Material and Formal Reasoning via LLM-Driven Neuro-Symbolic Refinement
Xin Quan, Marco Valentino, Danilo S. Carvalho +2
A persistent challenge in AI is the effective integration of material and formal inference - the former concerning the plausibility and contextual relevance of arguments, while the…
Improving Chain-of-Thought Reasoning via Quasi-Symbolic Abstractions
Leonardo Ranaldi, Marco Valentino, Andrè Freitas
Chain-of-Though (CoT) represents a common strategy for reasoning in Large Language Models (LLMs) by decomposing complex tasks into intermediate inference steps. However, explanatio…
Integrating Expert Knowledge into Logical Programs via LLMs
Franciszek Górski, Oskar Wysocki, Marco Valentino +1
This paper introduces ExKLoP, a novel framework designed to evaluate how effectively Large Language Models (LLMs) integrate expert knowledge into logical reasoning systems. This ca…
Autoformalization in the Wild: Assessing LLMs on Real-World Mathematical Definitions
Lan Zhang, Marco Valentino, Andre Freitas
Thanks to their linguistic capabilities, LLMs offer an opportunity to bridge the gap between informal mathematics and formal languages through autoformalization. However, it is sti…
Eliciting Critical Reasoning in Retrieval-Augmented Language Models via Contrastive Explanations
Leonardo Ranaldi, Marco Valentino, Andrè Freitas
Retrieval-augmented generation (RAG) has emerged as a critical mechanism in contemporary NLP to support Large Language Models(LLMs) in systematically accessing richer factual conte…
SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoning
Magdalena Wysocka, Danilo Carvalho, Oskar Wysocki +2
Syllogistic reasoning is crucial for Natural Language Inference (NLI). This capability is particularly significant in specialized domains such as biomedicine, where it can support…
Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference
Geonhee Kim, Marco Valentino, André Freitas
Recent studies on reasoning in language models (LMs) have sparked a debate on whether they can learn systematic inferential principles or merely exploit superficial patterns in the…
Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving
Xin Quan, Marco Valentino, Louise A. Dennis +1
Natural language explanations represent a proxy for evaluating explanation-based and multi-step Natural Language Inference (NLI) models. However, assessing the validity of explanat…
SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials
Mael Jullien, Marco Valentino, André Freitas
Large Language Models (LLMs) are at the forefront of NLP achievements but fall short in dealing with shortcut learning, factual inconsistency, and vulnerability to adversarial inpu…
A Differentiable Integer Linear Programming Solver for Explanation-Based Natural Language Inference
Mokanarangan Thayaparan, Marco Valentino, André Freitas
Integer Linear Programming (ILP) has been proposed as a formalism for encoding precise structural and semantic constraints for Natural Language Inference (NLI). However, traditiona…