106 citations · 167 across the 26 of their papers we have counts for
11 papers · 1 filter
Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines
Maximilian Böther, Josh Wills, Ties Robroek +14
Deterministic data loading is important for foundation model development: model researchers need confidence that differences they observe across costly ablations are caused by the…
Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter
Qinghao Hu, Shang Yang, Junxian Guo +7
The emergence of Large Language Models (LLMs) with strong reasoning capabilities marks a significant milestone, unlocking new frontiers in complex problem-solving. However, trainin…
Semantic-Aware Scheduling for GPU Clusters with Large Language Models
Zerui Wang, Qinghao Hu, Ana Klimovic +4
Deep learning (DL) schedulers are pivotal in optimizing resource allocation in GPU clusters, but operate with a critical limitation: they are largely blind to the semantic context…
Mixtera: A Data Plane for Foundation Model Training
Maximilian Böther, Xiaozhe Yao, Tolga Kerimoglu +3
State-of-the-art large language and vision models are trained over trillions of tokens that are aggregated from a large variety of sources. As training data collections grow, manua…
HashAttention: Semantic Sparsity for Faster Inference
Aditya Desai, Shuo Yang, Alejandro Cuadron +3
Leveraging long contexts is crucial for advanced AI systems, but attention computation poses a scalability challenge. While scaled dot-product attention (SDPA) exhibits token spars…
On Distributed Larger-Than-Memory Subset Selection With Pairwise Submodular Functions
Maximilian Böther, Abraham Sebastian, Pranjal Awasthi +2
Modern datasets span billions of samples, making training on all available data infeasible. Selecting a high quality subset helps in reducing training costs and enhancing model qua…
Modyn: Data-Centric Machine Learning Pipeline Orchestration
Maximilian Böther, Ties Robroek, Viktor Gsteiger +4
In real-world machine learning (ML) pipelines, datasets are continuously growing. Models must incorporate this new training data to improve generalization and adapt to potential di…
tf.data service: A Case for Disaggregating ML Input Data Processing
Andrew Audibert, Yang Chen, Dan Graur +3
Machine learning (ML) computations commonly execute on expensive specialized hardware, such as GPUs and TPUs, which provide high FLOPs and performance-per-watt. For cost efficiency…
SHiFT: An Efficient, Flexible Search Engine for Transfer Learning
Cedric Renggli, Xiaozhe Yao, Luka Kolar +3
Transfer learning can be seen as a data- and compute-efficient alternative to training models from scratch. The emergence of rich model repositories, such as TensorFlow Hub, enable…
Plumber: Diagnosing and Removing Performance Bottlenecks in Machine Learning Data Pipelines
Michael Kuchnik, Ana Klimovic, Jiri Simsa +2
Input pipelines, which ingest and transform input data, are an essential part of training Machine Learning (ML) models. However, it is challenging to implement efficient input pipe…
tf.data: A Machine Learning Data Processing Framework
Derek G. Murray, Jiri Simsa, Ana Klimovic +1
Training machine learning models requires feeding input data for models to ingest. Input pipelines for machine learning jobs are often challenging to implement efficiently as they…