66 citations · 102 across the 18 of their papers we have counts for
18 papers
Eleven Years of BRACIS: A Meta-Scientific Study of the Brazilian Conference on Intelligent Systems
Thales Sales Almeida, Giovana Kerche Bonás, Thiago Laitz +7
The Brazilian Conference on Intelligent Systems (BRACIS) is the main national venue for Artificial Intelligence research in Brazil, hosted by the Brazilian Computer Society since 2…
LLM-Based Persuasion Enables Guardrail Override in Frontier LLMs
Rodrigo Nogueira, Thales Sales Almeida, Giovana Kerche Bonás +7
Frontier assistant LLMs ship with strong guardrails: asked directly to write a persuasive essay denying the Holocaust, denying vaccine safety, defending flat-earth cosmology, argui…
Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks
Ramon Pires, Thales Sales Almeida, Celio Larcher Junior +6
Existing benchmarks for legal AI focus primarily on tasks where LLMs must produce legal arguments or documents, yet the capacity to \emph{judge} such arguments -- weighing competin…
Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6
Rankings produced by holistic LLM-as-a-judge scoring are sensitive to the bias of the chosen judge model. We show that switching to binary rubric scoring with multi-judge filtering…
Measuring Opinion Bias and Sycophancy via LLM-based Persuasion
Rodrigo Nogueira, Giovana Kerche Bonás, Thales Sales Almeida +7
Large language models increasingly shape the information people consume: they are embedded in search, consulted for professional advice, deployed as agents, and used as a first sto…
MARCA: A Checklist-Based Benchmark for Multilingual Web Search
Thales Sales Almeida, Giovana Kerche Bonás, Ramon Pires +6
Large language models (LLMs) are increasingly used as sources of information, yet their reliability depends on the ability to search the web, select relevant evidence, and synthesi…
CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context
Giovana Kerche Bonás, Roseval Malaquias Junior, Marcos Piau +6
We introduce CAPITU, a benchmark for evaluating instruction-following capabilities of Large Language Models (LLMs) in Brazilian Portuguese. Unlike existing benchmarks that focus on…
Sabiá-4 Technical Report
Thiago Laitz, Thales Sales Almeida, Hugo Abonizio +6
This technical report presents Sabiá-4 and Sabiazinho-4, a new generation of Portuguese language models with a focus on Brazilian Portuguese language. The models were developed thr…
PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese
Thales Sales Almeida, Ramon Pires, Hugo Abonizio +2
Large Language Models (LLMs) exhibit significant variations in performance across linguistic and cultural contexts, underscoring the need for systematic evaluation in diverse langu…
Automatic Legal Writing Evaluation of LLMs
Ramon Pires, Roseval Malaquias Junior, Rodrigo Nogueira
Despite the recent advances in Large Language Models, benchmarks for evaluating legal writing remain scarce due to the inherent complexity of assessing open-ended responses in this…
The interplay between domain specialization and model size
Roseval Malaquias Junior, Ramon Pires, Thales Sales Almeida +3
Scaling laws for language models have often focused on finding the optimal model size and token count for training from scratch. However, achieving this optimal balance requires si…
Sabiá-3 Technical Report
Hugo Abonizio, Thales Sales Almeida, Thiago Laitz +4
This report presents Sabiá-3, our new flagship language model, and Sabiazinho-3, a more cost-effective sibling. The models were trained on a large brazilian-centric corpus. Evaluat…
Sabiá-2: A New Generation of Portuguese Large Language Models
Thales Sales Almeida, Hugo Abonizio, Rodrigo Nogueira +1
We introduce Sabiá-2, a family of large language models trained on Portuguese texts. The models are evaluated on a diverse range of exams, including entry-level tests for Brazilian…
Juru: Legal Brazilian Large Language Model from Reputable Sources
Roseval Malaquias Junior, Ramon Pires, Roseli Romero +1
The high compute cost associated with pretraining large language models limits their research. Two strategies have emerged to address this issue: domain specialization and pretrain…
Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams
Ramon Pires, Thales Sales Almeida, Hugo Abonizio +1
Recent advancements in language models have showcased human-comparable performance in academic entrance exams. However, existing studies often overlook questions that require the i…
Sabiá: Portuguese Large Language Models
Ramon Pires, Hugo Abonizio, Thales Sales Almeida +1
As the capabilities of language models continue to advance, it is conceivable that "one-size-fits-all" model will remain as the main paradigm. For instance, given the vast number o…
Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Exams
Desnes Nunes, Ricardo Primi, Ramon Pires +2
The present study aims to explore the capabilities of Language Models (LMs) in tackling high-stakes multiple-choice tests, represented here by the Exame Nacional do Ensino Médio (E…
Sequence-to-Sequence Models for Extracting Information from Registration and Legal Documents
Ramon Pires, Fábio C. de Souza, Guilherme Rosa +2
A typical information extraction pipeline consists of token- or span-level classification models coupled with a series of pre- and post-processing scripts. In a production pipeline…