5 papers
Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines
Maximilian Böther, Josh Wills, Ties Robroek +14
Deterministic data loading is important for foundation model development: model researchers need confidence that differences they observe across costly ablations are caused by the…
20/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
DatologyAI, :, Siddharth Joshi +32
Data curation has shifted the quality-compute frontier for language-model and contrastive image-text pretraining, but its role for vision-language models (VLMs) is far less establi…
ÜberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset
DatologyAI, :, Aldo Gael Carranza +32
Multilinguality is a core capability for modern foundation models, yet training high-quality multilingual models remains challenging due to uneven data availability across language…
DatBench: Discriminative, Faithful, and Efficient VLM Evaluations
DatologyAI, :, Siddharth Joshi +30
Empirical evaluation serves as the primary compass guiding research progress in foundation models. Despite a large body of work focused on training frontier vision-language models…
BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining
DatologyAI, :, Pratyush Maini +28
Recent advances in large language model (LLM) pretraining have shown that simply scaling data quantity eventually leads to diminishing returns, hitting a data wall. In response, th…