21 citations · 47 across the 14 of their papers we have counts for
7 papers · 1 filter
Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison
Marianna Nezhurina, Harsh Raj, Jörg Franke +7
Reliable progress in language modeling depends on knowing whether a new dataset, architecture, or training recipe is genuinely better than existing alternatives. To foster reproduc…
Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets
Marianna Nezhurina, Tomer Porian, Giovanni Pucceti +4
In studies of transferable learning, scaling laws are obtained for various important foundation models to predict their properties and performance at larger scales. We show here ho…
OpenThoughts: Data Recipes for Reasoning Models
Etash Guha, Ryan Marten, Sedrick Keh +47
Reasoning models have made rapid progress on many benchmarks involving math, code, and science. Yet, there are still many open questions about the best training recipes for reasoni…
ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models
Adrian Mirza, Nawaf Alampara, Martiño Ríos-García +12
Foundation models have shown remarkable success across scientific domains, yet their impact in chemistry remains limited due to the absence of diverse, large-scale, high-quality da…
Learning in Compact Spaces with Approximately Normalized Transformer
Jörg K. H. Franke, Urs Spiegelhalter, Marianna Nezhurina +3
The successful training of deep neural networks requires addressing challenges such as overfitting, numerical instabilities leading to divergence, and increasing variance in the re…
DataComp-LM: In search of the next generation of training sets for language models
Jeffrey Li, Alex Fang, Georgios Smyrnis +56
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardize…
Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models
Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti +1
Large Language Models (LLMs) are often described as instances of foundation models that possess strong generalization obeying scaling laws, and therefore transfer robustly across v…