3 citations · 8 across the 24 of their papers we have counts for
27 papers
MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support
John Mendonça, Daniel Tschernutter, José Pombal +6
With people increasingly turning to AI chatbots for mental health support, the need to evaluate these tools is becoming more pressing. Recent benchmarks attempt to mimic a realisti…
DEPICT: Scoring Text-to-Image Alignment by Answer Agreement
Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes +2
Image-text alignment is a core problem in computer vision with applications in caption evaluation, hallucination detection, data curation, and the benchmarking of text-to-image (T2…
IndicQE-APE: A Consolidated Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages
Diptesh Kanojia, Archchana Sindhujan, Sourabh Deoghare +14
Indic quality estimation (QE) and automatic post-editing (APE) data is spread across separate releases, so no single resource supports training and evaluation across tasks and lang…
Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy
Afonso Baldo, Hugo Pitorro, Areti Vassilopoulos +5
As the demand for mental health services outpaces clinical workforce capacity, users are increasingly turning to large language models (LLMs) for accessible and private support. Al…
Risk Governance for Generative AI Mental Health Support: A Multi-Turn Safety Architecture
Anabela C. Areias, Catarina Botelho, António Farinhas +7
Large language models (LLMs) are increasingly used for emotional support despite lacking mechanisms to safely govern evolving mental health risk. Existing safety approaches primari…
Can Vision Language Models Judge Action Quality? An Empirical Evaluation
Miguel Monte e Freitas, Rui Henriques, Ricardo Rei +1
Action Quality Assessment (AQA) has broad applications in physical therapy, sports coaching, and competitive judging. Although Vision Language Models (VLMs) hold considerable promi…
Self-Preference Bias in Rubric-Based Evaluation of Large Language Models
José Pombal, Ricardo Rei, André F. T. Martins
LLM-as-a-judge has become the de facto approach for evaluating LLM outputs. However, judges are known to exhibit self-preference bias (SPB): they tend to favor outputs produced by…
EuroLLM-22B: Technical Report
Miguel Moura Ramos, Duarte M. Alves, Hippolyte Gisserot-Boukhlef +15
This report presents EuroLLM-22B, a large language model trained from scratch to support the needs of European citizens by covering all 24 official European Union languages and 11…
MindGuard: Guardrail Classifiers for Multi-Turn Mental Health Support
António Farinhas, Nuno M. Guerreiro, José Pombal +6
Large language models are increasingly used for mental health support, yet their conversational coherence alone does not ensure clinical appropriateness. Existing general-purpose s…
MindEval: Benchmarking Language Models on Multi-turn Mental Health Support
José Pombal, Maya D'Eon, Nuno M. Guerreiro +3
Demand for mental health support through AI chatbots is surging, though current systems present several limitations, like sycophancy or overvalidation, and reinforcement of maladap…
Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs
Ricardo Rei, Nuno M. Guerreiro, José Pombal +4
Fine-tuning pretrained LLMs has been shown to be an effective strategy for reaching state-of-the-art performance on specific tasks like machine translation. However, this process o…
EuroLLM-9B: Technical Report
Pedro Henrique Martins, João Alves, Patrick Fernandes +14
This report presents EuroLLM-9B, a large language model trained from scratch to support the needs of European citizens by covering all 24 official European Union languages and 11 a…
M-Prometheus: A Suite of Open Multilingual LLM Judges
José Pombal, Dongkeun Yoon, Patrick Fernandes +5
The use of language models for automatically evaluating long-form text (LLM-as-a-judge) is becoming increasingly common, yet most LLM judges are optimized exclusively for English,…
Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models
José Pombal, Nuno M. Guerreiro, Ricardo Rei +1
As language models improve and become capable of performing more complex tasks across modalities, evaluating them automatically becomes increasingly challenging. Developing strong…
XL-Suite: Cross-Lingual Synthetic Training and Evaluation Data for Open-Ended Generation
Vivek Iyer, Pinzhen Chen, Ricardo Rei +1
Cross-lingual open-ended generation - responding in a language different from that of the query - is an important yet understudied problem. This work proposes XL-Instruct, a novel…
Adding Chocolate to Mint: Mitigating Metric Interference in Machine Translation
José Pombal, Nuno M. Guerreiro, Ricardo Rei +1
As automatic metrics become increasingly stronger and widely adopted, the risk of unintentionally "gaming the metric" during model development rises. This issue is caused by metric…
EuroBERT: Scaling Multilingual Encoders for European Languages
Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves +16
General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models. Despite their w…
Translate Smart, not Hard: Cascaded Translation Systems with Quality-Aware Deferral
António Farinhas, Nuno M. Guerreiro, Sweta Agrawal +2
Larger models often outperform smaller ones but come with high computational costs. Cascading offers a potential solution. By default, it uses smaller models and defers only some i…
Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation
Sweta Agrawal, José G. C. de Souza, Ricardo Rei +5
Alignment with human preferences is an important step in developing accurate and safe large language models. This is no exception in machine translation (MT), where better handling…
Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis
Hippolyte Gisserot-Boukhlef, Ricardo Rei, Emmanuel Malherbe +3
Neural metrics for machine translation (MT) evaluation have become increasingly prominent due to their superior correlation with human judgments compared to traditional lexical met…
EuroLLM: Multilingual Language Models for Europe
Pedro Henrique Martins, Patrick Fernandes, João Alves +12
The quality of open-weight LLMs has seen significant improvement, yet they remain predominantly focused on English. In this paper, we introduce the EuroLLM project, aimed at develo…
Thesis proposal: Are We Losing Textual Diversity to Natural Language Processing?
Josef Jon
This thesis argues that the currently widely used Natural Language Processing algorithms possibly have various limitations related to the properties of the texts they handle and pr…
xTower: A Multilingual LLM for Explaining and Correcting Translation Errors
Marcos Treviso, Nuno M. Guerreiro, Sweta Agrawal +7
While machine translation (MT) systems are achieving increasingly strong performance on benchmarks, they often produce translations with errors and anomalies. Understanding these e…
QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation
Gonçalo R. A. Faria, Sweta Agrawal, António Farinhas +3
An important challenge in machine translation (MT) is to generate high-quality and diverse translations. Prior work has shown that the estimated likelihood from the MT model correl…