Publications (11)
Luxical: High-Speed Lexical-Dense Text Embeddings
DatologyAI, :, Luke Merrick +31
Frontier language model quality increasingly hinges on our ability to organize web-scale text corpora for training. Today's dominant tools trade off speed and flexibility: lexical…
ÃberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset
DatologyAI, :, Aldo Gael Carranza +32
Multilinguality is a core capability for modern foundation models, yet training high-quality multilingual models remains challenging due to uneven data availability across language…
DatBench: Discriminative, Faithful, and Efficient VLM Evaluations
DatologyAI, :, Siddharth Joshi +30
Empirical evaluation serves as the primary compass guiding research progress in foundation models. Despite a large body of work focused on training frontier vision-language models…
The Explanation Game: Explaining Machine Learning Models Using Shapley Values
Luke Merrick, Ankur Taly
A number of techniques have been proposed to explain a machine learning model's prediction by attributing it to the corresponding input features. Popular among these are techniques…
Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models
Luke Merrick, Danmei Xu, Gaurav Nuti +1
This report describes the training dataset creation and recipe behind the family of \texttt{arctic-embed} text embedding models (a set of five models ranging from 22 to 334 million…
BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining
DatologyAI, :, Pratyush Maini +28
Recent advances in large language model (LLM) pretraining have shown that simply scaling data quantity eventually leads to diminishing returns, hitting a data wall. In response, th…
20/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
DatologyAI, :, Siddharth Joshi +32
Data curation has shifted the quality-compute frontier for language-model and contrastive image-text pretraining, but its role for vision-language models (VLMs) is far less establi…
Arctic-Embed 2.0: Multilingual Retrieval Without Compromise
Puxuan Yu, Luke Merrick, Gaurav Nuti +1
This paper presents the training methodology of Arctic-Embed 2.0, a set of open-source text embedding models built for accurate and efficient multilingual retrieval. While prior wo…
Embedding And Clustering Your Data Can Improve Contrastive Pretraining
Luke Merrick
Recent studies of large-scale contrastive pretraining in the text embedding domain show that using single-source minibatches, rather than mixed-source minibatches, can substantiall…
The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data
Christina Baek, Ricardo Pio Monti, David Schwab +31
Real-world model deployments demand strong performance on narrow domains where data is often scarce. Typically, practitioners finetune models to specialize them, but this risks ove…
Randomized Ablation Feature Importance
Luke Merrick
Given a model that predicts a target from a vector of input features , we seek to measure the importance of each feature with respect to th…