1.1k citations · 2.8k across the 33 of their papers we have counts for
28 papers · 1 filter
This Time is Different: An Observability Perspective on Time Series Foundation Models
Ben Cohen, Emaad Khwaja, Youssef Doubli +16
We introduce Toto, a time series forecasting foundation model with 151 million parameters. Toto uses a modern decoder-only architecture coupled with architectural innovations desig…
Analyzing Similarity Metrics for Data Selection for Language Model Pretraining
Dylan Sam, Ayan Chakrabarti, Afshin Rostamizadeh +3
Measuring similarity between training examples is critical for curating high-quality and diverse pretraining datasets for language models. However, similarity is typically computed…
A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs
Ankit Singh Rawat, Veeranjaneyulu Sadhanala, Afshin Rostamizadeh +12
A primary challenge in large language model (LLM) development is their onerous pre-training cost. Typically, such pre-training involves optimizing a self-supervised objective (such…
SoftSRV: Learn to Generate Targeted Synthetic Data
Giulia DeSalvo, Jean-Fracois Kagy, Lazaros Karydas +2
We present a novel framework, SoftSRV, that is used to generate targeted synthetic fine-tuning data for improving task-specific model performance. Given a sample from a target dist…
SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced Token Detection
Ke Ye, Heinrich Jiang, Afshin Rostamizadeh +6
Pre-training large language models is known to be extremely resource intensive and often times inefficient, under-utilizing the information encapsulated in the training text sequen…
Leveraging Importance Weights in Subset Selection
Gui Citovsky, Giulia DeSalvo, Sanjiv Kumar +3
We present a subset selection algorithm designed to work with arbitrary model families in a practical batch setting. In such a setting, an algorithm can sample examples one at a ti…