47 citations · 54 across the 8 of their papers we have counts for
10 papers
Analyzing Similarity Metrics for Data Selection for Language Model Pretraining
Dylan Sam, Ayan Chakrabarti, Afshin Rostamizadeh +3
Measuring similarity between training examples is critical for curating high-quality and diverse pretraining datasets for language models. However, similarity is typically computed…
GIST: Greedy Independent Set Thresholding for Max-Min Diversification with Submodular Utility
Matthew Fahrbach, Srikumar Ramalingam, Morteza Zadimoghaddam +3
This work studies a novel subset selection problem called max-min diversification with monotone submodular utility (), which has a wide range of applications in mach…
SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced Token Detection
Ke Ye, Heinrich Jiang, Afshin Rostamizadeh +6
Pre-training large language models is known to be extremely resource intensive and often times inefficient, under-utilizing the information encapsulated in the training text sequen…
Leveraging Importance Weights in Subset Selection
Gui Citovsky, Giulia DeSalvo, Sanjiv Kumar +3
We present a subset selection algorithm designed to work with arbitrary model families in a practical batch setting. In such a setting, an algorithm can sample examples one at a ti…
Batch Active Learning at Scale
Gui Citovsky, Giulia DeSalvo, Claudio Gentile +4
The ability to train complex and highly effective models often requires an abundance of training data, which can easily become a bottleneck in cost, time, and computational resourc…
Scaling Hierarchical Agglomerative Clustering to Billion-sized Datasets
Baris Sumengen, Anand Rajagopalan, Gui Citovsky +6
Hierarchical Agglomerative Clustering (HAC) is one of the oldest but still most widely used clustering methods. However, HAC is notoriously hard to scale to large data sets as the…