106 citations · 225 across the 3 of their papers we have counts for
3 papers
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Hugo Laurençon, Lucile Saulnier, Thomas Wang +51
As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop,…
SantaCoder: don't reach for the stars!
Loubna Ben Allal, Raymond Li, Denis Kocetkov +38
The BigCode project is an open-scientific collaboration working on the responsible development of large language models for code. This tech report describes the progress of the col…
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky +8
What are the units of text that we want to model? From bytes to multi-word expressions, text can be analyzed and generated at many granularities. Until recently, most natural langu…