4 citations · 12 across the 26 of their papers we have counts for
25 papers · 1 filter
PRISM: Transfer and Limits of Activation Interventions in Tool-Calling Language Models
Giovana Kerche Bonás, Thales Sales Almeida, Marcelo da Silva Reis +2
Recent literature shows that unsafe tool calls can often accompany textual refusals in language models. To investigate this hidden vulnerability, we introduce PRISM, a framework fo…
BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams
João Guilherme Alves Santos, Giovana Kerche Bonás, Thiago Laitz +2
Although Large Language Models (LLMs) excel in many tasks, their assessment in Portuguese has received less attention, particularly for open-ended, discursive tasks that demand dee…
moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT
Thiago Laitz, Thales Sales Almeida, João Guilherme Alves Santos +1
Encoder-only transformer models remain essential for production NLP pipelines. We introduce moBERTo, a Portuguese adaptation of ModernBERT obtained through continued pretraining of…
LLM-Based Persuasion Enables Guardrail Override in Frontier LLMs
Rodrigo Nogueira, Thales Sales Almeida, Giovana Kerche Bonás +7
Frontier assistant LLMs ship with strong guardrails: asked directly to write a persuasive essay denying the Holocaust, denying vaccine safety, defending flat-earth cosmology, argui…
Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks
Ramon Pires, Thales Sales Almeida, Celio Larcher Junior +6
Existing benchmarks for legal AI focus primarily on tasks where LLMs must produce legal arguments or documents, yet the capacity to \emph{judge} such arguments -- weighing competin…
Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6
Rankings produced by holistic LLM-as-a-judge scoring are sensitive to the bias of the chosen judge model. We show that switching to binary rubric scoring with multi-judge filtering…
Measuring Opinion Bias and Sycophancy via LLM-based Persuasion
Rodrigo Nogueira, Giovana Kerche Bonás, Thales Sales Almeida +7
Large language models increasingly shape the information people consume: they are embedded in search, consulted for professional advice, deployed as agents, and used as a first sto…
MARCA: A Checklist-Based Benchmark for Multilingual Web Search
Thales Sales Almeida, Giovana Kerche Bonás, Ramon Pires +6
Large language models (LLMs) are increasingly used as sources of information, yet their reliability depends on the ability to search the web, select relevant evidence, and synthesi…
Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining
Thales Sales Almeida, Rodrigo Nogueira, Hélio Pedrini
Synthetic data generation through document rewriting has emerged as a promising technique for improving language model pretraining, yet most studies focus on English and do not sys…
CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context
Giovana Kerche Bonás, Roseval Malaquias Junior, Marcos Piau +6
We introduce CAPITU, a benchmark for evaluating instruction-following capabilities of Large Language Models (LLMs) in Brazilian Portuguese. Unlike existing benchmarks that focus on…
Sabiá-4 Technical Report
Thiago Laitz, Thales Sales Almeida, Hugo Abonizio +6
This technical report presents Sabiá-4 and Sabiazinho-4, a new generation of Portuguese language models with a focus on Brazilian Portuguese language. The models were developed thr…
Curió-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretraining
Thales Sales Almeida, Rodrigo Nogueira, Hélio Pedrini
Continued pretraining extends a language model's capabilities by further exposing it to additional data, often tailored to a specific linguistic or domain context. This strategy ha…
Ticket-Bench: A Kickoff for Multilingual and Regionalized Agent Evaluation
Thales Sales Almeida, João Guilherme Alves Santos, Thiago Laitz +1
Large language models (LLMs) are increasingly deployed as task-oriented agents, where success depends on their ability to generate accurate function calls under realistic, multilin…
BRoverbs -- Measuring how much LLMs understand Portuguese proverbs
Thales Sales Almeida, Giovana Kerche Bonás, João Guilherme Alves Santos
Large Language Models (LLMs) exhibit significant performance variations depending on the linguistic and cultural context in which they are applied. This disparity signals the neces…
Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora
Thales Sales Almeida, Rodrigo Nogueira, Helio Pedrini
The performance of large language models (LLMs) is deeply influenced by the quality and composition of their training data. While much of the existing work has centered on English,…
BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning
João Guilherme Alves Santos, Giovana Kerche Bonás, Thales Sales Almeida
With the growing capabilities of Large Language Models (LLMs), there is an increasing need for robust evaluation methods, especially in multilingual and non-English contexts. We pr…
Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime
Hugo Abonizio, Thales Almeida, Roberto Lotufo +1
Large language models (LLMs) often require vast amounts of text to effectively acquire new knowledge. While continuing pre-training on large corpora or employing retrieval-augmente…
TiEBe: Tracking Language Model Recall of Notable Worldwide Events Through Time
Thales Sales Almeida, Giovana Kerche Bonás, João Guilherme Alves Santos +2
As the knowledge landscape evolves and large language models (LLMs) become increasingly widespread, there is a growing need to keep these models updated with current events. While…
The interplay between domain specialization and model size
Roseval Malaquias Junior, Ramon Pires, Thales Sales Almeida +3
Scaling laws for language models have often focused on finding the optimal model size and token count for training from scratch. However, achieving this optimal balance requires si…
Sabiá-3 Technical Report
Hugo Abonizio, Thales Sales Almeida, Thiago Laitz +4
This report presents Sabiá-3, our new flagship language model, and Sabiazinho-3, a more cost-effective sibling. The models were trained on a large brazilian-centric corpus. Evaluat…
SurveySum: A Dataset for Summarizing Multiple Scientific Articles into a Survey Section
Leandro Carísio Fernandes, Gustavo Bartz Guedes, Thiago Soares Laitz +4
Document summarization is a task to shorten texts into concise and informative summaries. This paper introduces a novel dataset designed for summarizing multiple scientific article…
Measuring Cross-lingual Transfer in Bytes
Leandro Rodrigues de Souza, Thales Sales Almeida, Roberto Lotufo +1
Multilingual pretraining has been a successful solution to the challenges posed by the lack of resources for languages. These models can transfer knowledge to target languages with…
Sabiá-2: A New Generation of Portuguese Large Language Models
Thales Sales Almeida, Hugo Abonizio, Rodrigo Nogueira +1
We introduce Sabiá-2, a family of large language models trained on Portuguese texts. The models are evaluated on a diverse range of exams, including entry-level tests for Brazilian…
Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams
Ramon Pires, Thales Sales Almeida, Hugo Abonizio +1
Recent advancements in language models have showcased human-comparable performance in academic entrance exams. However, existing studies often overlook questions that require the i…