6 citations · 7 across the 5 of their papers we have counts for
5 papers
CAT: A Metric-Driven Framework for Analyzing the Consistency-Accuracy Relation of LLMs under Controlled Input Variations
Paulo Cavalin, Cassia Sanctos, Marcelo Grave +2
We introduce \textsc{CAT}, a framework designed to evaluate and visualize the \emph{interplay} of \emph{accuracy} and \emph{response consistency} of Large Language Models (LLMs) un…
Improving Score Reliability of Multiple Choice Benchmarks with Consistency Evaluation and Altered Answer Choices
Paulo Cavalin, Cassia Sanctos, Marcelo Grave +2
In this work we present the Consistency-Rebalanced Accuracy (CoRA) metric, improving the reliability of Large Language Model (LLM) scores computed on multiple choice (MC) benchmark…
The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarks
Claudio Pinhanez, Paulo Cavalin, Cassia Sanctos +2
This work explores the consistency of small LLMs (2B-8B parameters) in answering multiple times the same question. We present a study on known, open-source LLMs responding to 10 re…
Harnessing the Power of Artificial Intelligence to Vitalize Endangered Indigenous Languages: Technologies and Experiences
Claudio Pinhanez, Paulo Cavalin, Luciana Storto +9
Since 2022 we have been exploring application areas and technologies in which Artificial Intelligence (AI) and modern Natural Language Processing (NLP), such as Large Language Mode…
Creating an African American-Sounding TTS: Guidelines, Technical Challenges,and Surprising Evaluations
Claudio Pinhanez, Raul Fernandez, Marcelo Grave +2
Representations of AI agents in user interfaces and robotics are predominantly White, not only in terms of facial and skin features, but also in the synthetic voices they use. In t…