6 citations · 15 across the 14 of their papers we have counts for
17 papers
MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support
John Mendonça, Daniel Tschernutter, José Pombal +6
With people increasingly turning to AI chatbots for mental health support, the need to evaluate these tools is becoming more pressing. Recent benchmarks attempt to mimic a realisti…
SEQUOR: A Multi-Turn Benchmark for Realistic Constraint Following
Beatriz Canaverde, Duarte M. Alves, José Pombal +2
In a conversation, a helpful assistant must reliably follow user directives, even as they refine, modify, or contradict earlier requests. Yet most instruction-following benchmarks…
Self-Preference Bias in Rubric-Based Evaluation of Large Language Models
José Pombal, Ricardo Rei, André F. T. Martins
LLM-as-a-judge has become the de facto approach for evaluating LLM outputs. However, judges are known to exhibit self-preference bias (SPB): they tend to favor outputs produced by…
EuroLLM-22B: Technical Report
Miguel Moura Ramos, Duarte M. Alves, Hippolyte Gisserot-Boukhlef +15
This report presents EuroLLM-22B, a large language model trained from scratch to support the needs of European citizens by covering all 24 official European Union languages and 11…
MindGuard: Guardrail Classifiers for Multi-Turn Mental Health Support
António Farinhas, Nuno M. Guerreiro, José Pombal +6
Large language models are increasingly used for mental health support, yet their conversational coherence alone does not ensure clinical appropriateness. Existing general-purpose s…
MindEval: Benchmarking Language Models on Multi-turn Mental Health Support
José Pombal, Maya D'Eon, Nuno M. Guerreiro +3
Demand for mental health support through AI chatbots is surging, though current systems present several limitations, like sycophancy or overvalidation, and reinforcement of maladap…
Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs
Ricardo Rei, Nuno M. Guerreiro, José Pombal +4
Fine-tuning pretrained LLMs has been shown to be an effective strategy for reaching state-of-the-art performance on specific tasks like machine translation. However, this process o…
EuroLLM-9B: Technical Report
Pedro Henrique Martins, João Alves, Patrick Fernandes +14
This report presents EuroLLM-9B, a large language model trained from scratch to support the needs of European citizens by covering all 24 official European Union languages and 11 a…
M-Prometheus: A Suite of Open Multilingual LLM Judges
José Pombal, Dongkeun Yoon, Patrick Fernandes +5
The use of language models for automatically evaluating long-form text (LLM-as-a-judge) is becoming increasingly common, yet most LLM judges are optimized exclusively for English,…
Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models
José Pombal, Nuno M. Guerreiro, Ricardo Rei +1
As language models improve and become capable of performing more complex tasks across modalities, evaluating them automatically becomes increasingly challenging. Developing strong…
Adding Chocolate to Mint: Mitigating Metric Interference in Machine Translation
José Pombal, Nuno M. Guerreiro, Ricardo Rei +1
As automatic metrics become increasingly stronger and widely adopted, the risk of unintentionally "gaming the metric" during model development rises. This issue is caused by metric…
A Context-aware Framework for Translation-mediated Conversations
José Pombal, Sweta Agrawal, Patrick Fernandes +2
Automatic translation systems offer a powerful solution to bridge language barriers in scenarios where participants do not share a common language. However, these systems can intro…
EuroLLM: Multilingual Language Models for Europe
Pedro Henrique Martins, Patrick Fernandes, João Alves +12
The quality of open-weight LLMs has seen significant improvement, yet they remain predominantly focused on English. In this paper, we introduce the EuroLLM project, aimed at develo…
xTower: A Multilingual LLM for Explaining and Correcting Translation Errors
Marcos Treviso, Nuno M. Guerreiro, Sweta Agrawal +7
While machine translation (MT) systems are achieving increasingly strong performance on benchmarks, they often produce translations with errors and anomalies. Understanding these e…
Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
Duarte M. Alves, José Pombal, Nuno M. Guerreiro +10
While general-purpose large language models (LLMs) demonstrate proficiency on multiple tasks within the domain of translation, approaches based on open LLMs are competitive only wh…
Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning
Duarte M. Alves, Nuno M. Guerreiro, João Alves +5
Large language models (LLMs) are a promising avenue for machine translation (MT). However, current LLM-based MT systems are brittle: their effectiveness highly depends on the choic…
Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task
Ricardo Rei, Nuno M. Guerreiro, José Pombal +5
We present the joint contribution of Unbabel and Instituto Superior Técnico to the WMT 2023 Shared Task on Quality Estimation (QE). Our team participated on all tasks: sentence- an…