Tucano: Advancing Neural Text Generation for Portuguese

Abstract

Significant advances have been made in natural language processing in recent years. However, our current deep learning approach to language modeling requires substantial resources in terms of data and computation. One of the side effects of this data-hungry paradigm is the current schism between languages, separating those considered high-resource, where most of the development happens and resources are available, and the low-resource ones, which struggle to attain the same level of performance and autonomy. This study aims to introduce a new set of resources to stimulate the future development of neural text generation in Portuguese. In this work, we document the development of GigaVerbo, a concatenation of deduplicated Portuguese text corpora amounting to 200 billion tokens. Via this corpus, we trained a series of decoder-transformers named Tucano. Our models perform equal or superior to other Portuguese and multilingual language models of similar size in several Portuguese benchmarks. The evaluation of our models also reveals that model performance on many currently available benchmarks used by the Portuguese NLP community has little to no correlation with the scaling of token ingestion during training, highlighting the limitations of such evaluations when it comes to the assessment of Portuguese generative language models. All derivatives of our study are openly released on GitHub and Hugging Face. See https://nkluge-correa.github.io/Tucano/

References in corpus (34)

• Sequence to Sequence Learning with Neural Networks
• Decoupled Weight Decay Regularization
• Training language models to follow instructions with human feedback
• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• On the Opportunities and Risks of Foundation Models
• PaLM: Scaling Language Modeling with Pathways
• Visual Instruction Tuning
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• Root Mean Square Layer Normalization
• Textbooks Are All You Need
• A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
• The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
• TinyLlama: An Open-Source Small Language Model
• AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
• Textbooks Are All You Need II: phi-1.5 technical report
• Energy Usage Reports: Environmental awareness as part of algorithmic accountability
• Scaling Data-Constrained Language Models
• Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster
• TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese
• Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• PolyLM: An Open Source Polyglot Large Language Model
• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
• Orca-Math: Unlocking the potential of SLMs in Grade School Math
• A New Massive Multilingual Dataset for High-Performance Language Technologies
• Sabiá-2: A New Generation of Portuguese Large Language Models
• Cabrita: closing the gap for foreign languages
• Juru: Legal Brazilian Large Language Model from Reputable Sources