9 papers
MATCHA: Matching Text via Contrastive Semantic Alignment
Siran Li, Ece Sena Etoglu, Carsten Eickhoff +1
Reliable evaluation is essential for understanding large language model (LLM) performance, yet today's go-to metrics, namely token-overlap scores (e.g., ROUGE) and embedding-based…
Beyond Multiple Choice: Evaluating Steering Vectors for Summarization
Joschka Braun, Carsten Eickhoff, Seyed Ali Bahrainian
Steering vectors are a lightweight method for controlling text properties by adding a learned bias to language model activations at inference time. While predominantly studied for…
When Silence Is Golden: Can LLMs Learn to Abstain in Temporal QA and Beyond?
Xinyu Zhou, Chang Jin, Carsten Eickhoff +2
Large language models (LLMs) rarely admit uncertainty, often producing fluent but misleading answers, rather than abstaining (i.e., refusing to answer). This weakness is even evide…
Navigating through the hidden embedding space: steering LLMs to improve mental health assessment
Federico Ravenda, Seyed Ali Bahrainian, Andrea Raballo +1
The rapid evolution of Large Language Models (LLMs) is transforming AI, opening new opportunities in sensitive and high-impact areas such as Mental Health (MH). Yet, despite these…
Benchmarking is Broken -- Don't Let AI be its Own Judge
Zerui Cheng, Stella Wohnig, Ruchika Gupta +13
The meteoric rise of AI, with its rapidly expanding market capitalization, presents both transformative opportunities and critical challenges. Chief among these is the urgent need…
Logit Reweighting for Topic-Focused Summarization
Joschka Braun, Bálint Mucsányi, Seyed Ali Bahrainian
Generating abstractive summaries that adhere to a specific topic remains a significant challenge for language models. While standard approaches, such as fine-tuning, are resource-i…