66 citations · 140 across the 23 of their papers we have counts for
18 papers · 1 filter
LLM-Based Persuasion Enables Guardrail Override in Frontier LLMs
Rodrigo Nogueira, Thales Sales Almeida, Giovana Kerche Bonás +7
Frontier assistant LLMs ship with strong guardrails: asked directly to write a persuasive essay denying the Holocaust, denying vaccine safety, defending flat-earth cosmology, argui…
Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks
Ramon Pires, Thales Sales Almeida, Celio Larcher Junior +6
Existing benchmarks for legal AI focus primarily on tasks where LLMs must produce legal arguments or documents, yet the capacity to \emph{judge} such arguments -- weighing competin…
Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6
Rankings produced by holistic LLM-as-a-judge scoring are sensitive to the bias of the chosen judge model. We show that switching to binary rubric scoring with multi-judge filtering…
Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines
Hugo Abonizio, Filipe Rocha Lopes, Roberto Lotufo +1
Brazil's Unified Health System (SUS) relies on official clinical guidelines that define diagnostic criteria, treatments, dosages, and monitoring procedures for over 200 million cit…
Measuring Opinion Bias and Sycophancy via LLM-based Persuasion
Rodrigo Nogueira, Giovana Kerche Bonás, Thales Sales Almeida +7
Large language models increasingly shape the information people consume: they are embedded in search, consulted for professional advice, deployed as agents, and used as a first sto…
MARCA: A Checklist-Based Benchmark for Multilingual Web Search
Thales Sales Almeida, Giovana Kerche Bonás, Ramon Pires +6
Large language models (LLMs) are increasingly used as sources of information, yet their reliability depends on the ability to search the web, select relevant evidence, and synthesi…
CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context
Giovana Kerche Bonás, Roseval Malaquias Junior, Marcos Piau +6
We introduce CAPITU, a benchmark for evaluating instruction-following capabilities of Large Language Models (LLMs) in Brazilian Portuguese. Unlike existing benchmarks that focus on…
Sabiá-4 Technical Report
Thiago Laitz, Thales Sales Almeida, Hugo Abonizio +6
This technical report presents Sabiá-4 and Sabiazinho-4, a new generation of Portuguese language models with a focus on Brazilian Portuguese language. The models were developed thr…
PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese
Thales Sales Almeida, Ramon Pires, Hugo Abonizio +2
Large Language Models (LLMs) exhibit significant variations in performance across linguistic and cultural contexts, underscoring the need for systematic evaluation in diverse langu…
Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime
Hugo Abonizio, Thales Almeida, Roberto Lotufo +1
Large language models (LLMs) often require vast amounts of text to effectively acquire new knowledge. While continuing pre-training on large corpora or employing retrieval-augmente…
TiEBe: Tracking Language Model Recall of Notable Worldwide Events Through Time
Thales Sales Almeida, Giovana Kerche Bonás, João Guilherme Alves Santos +2
As the knowledge landscape evolves and large language models (LLMs) become increasingly widespread, there is a growing need to keep these models updated with current events. While…
Sabiá-3 Technical Report
Hugo Abonizio, Thales Sales Almeida, Thiago Laitz +4
This report presents Sabiá-3, our new flagship language model, and Sabiazinho-3, a more cost-effective sibling. The models were trained on a large brazilian-centric corpus. Evaluat…
Sabiá-2: A New Generation of Portuguese Large Language Models
Thales Sales Almeida, Hugo Abonizio, Rodrigo Nogueira +1
We introduce Sabiá-2, a family of large language models trained on Portuguese texts. The models are evaluated on a diverse range of exams, including entry-level tests for Brazilian…
Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams
Ramon Pires, Thales Sales Almeida, Hugo Abonizio +1
Recent advancements in language models have showcased human-comparable performance in academic entrance exams. However, existing studies often overlook questions that require the i…
Sabiá: Portuguese Large Language Models
Ramon Pires, Hugo Abonizio, Thales Sales Almeida +1
As the capabilities of language models continue to advance, it is conceivable that "one-size-fits-all" model will remain as the main paradigm. For instance, given the vast number o…
MonoByte: A Pool of Monolingual Byte-level Language Models
Hugo Abonizio, Leandro Rodrigues de Souza, Roberto Lotufo +1
The zero-shot cross-lingual ability of models pretrained on multilingual and even monolingual corpora has spurred many hypotheses to explain this intriguing empirical result. Howev…
Billions of Parameters Are Worth More Than In-domain Training Data: A case study in the Legal Case Entailment Task
Guilherme Moraes Rosa, Luiz Bonifacio, Vitor Jeronymo +3
Recent work has shown that language models scaled to billions of parameters, such as GPT-3, perform remarkably well in zero-shot and few-shot scenarios. In this work, we experiment…
InPars: Data Augmentation for Information Retrieval using Large Language Models
Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee +1
The information retrieval community has recently witnessed a revolution due to large pretrained transformer models. Another key ingredient for this revolution was the MS MARCO data…