7 papers · 1 filter
FActBench: A Benchmark for Fine-grained Automatic Evaluation of LLM-Generated Text in the Medical Domain
Anum Afzal, Juraj Vladika, Florian Matthes
Large Language Models tend to struggle when dealing with specialized domains. While all aspects of evaluation hold importance, factuality is the most critical one. Similarly, relia…
Can Smaller LLMs do better? Unlocking Cross-Domain Potential through Parameter-Efficient Fine-Tuning for Text Summarization
Anum Afzal, Mehul Kumawat, Florian Matthes
Large Language Models (LLMs), being generic task solvers, are versatile. However, despite the vast amount of data they are trained on, there are speculations about their adaptation…
MedSEBA: Synthesizing Evidence-Based Answers Grounded in Evolving Medical Literature
Juraj Vladika, Florian Matthes
In the digital age, people often turn to the Internet in search of medical advice and recommendations. With the increasing volume of online content, it has become difficult to dist…
Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion
Anum Afzal, Florian Matthes, Gal Chechik +1
We investigate whether the success of a zero-shot Chain-of-Thought (CoT) process can be predicted before completion. We discover that a probing classifier, based on LLM representat…
JaccDiv: A Metric and Benchmark for Quantifying Diversity of Generated Marketing Text in the Music Industry
Anum Afzal, Alexandre Mercier, Florian Matthes
Online platforms are increasingly interested in using Data-to-Text technologies to generate content and help their users. Unfortunately, traditional generative methods often fall i…
AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarization
Anum Afzal, Ribin Chalumattu, Florian Matthes +1
Despite the advances in the abstractive summarization task using Large Language Models (LLM), there is a lack of research that asses their abilities to easily adapt to different do…