203 citations · 593 across the 26 of their papers we have counts for
11 papers · 1 filter
Co-design Hardware and Algorithm for Vector Search
Wenqi Jiang, Shigang Li, Yu Zhu +8
Vector search has emerged as the foundation for large-scale information retrieval and machine learning systems, with search engines like Google and Bing processing tens of thousand…
ASDL: A Unified Interface for Gradient Preconditioning in PyTorch
Kazuki Osawa, Satoki Ishikawa, Rio Yokota +2
Gradient preconditioning is a key technique to integrate the second-order information into gradients for improving and extending gradient-based learning algorithms. In deep learnin…
PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices
Kazuki Osawa, Shigang Li, Torsten Hoefler
Pipeline parallelism enables efficient training of Large Language Models (LLMs) on large-scale distributed accelerator clusters. Yet, pipeline bubbles during startup and tear-down…
A Data-Centric Optimization Framework for Machine Learning
Oliver Rausch, Tal Ben-Nun, Nikoli Dryden +3
Rapid progress in deep learning is leading to a diverse set of quickly changing models, with a dramatically growing demand for compute. However, as frameworks specialize performanc…
Data Movement Is All You Need: A Case Study on Optimizing Transformers
Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun +2
Transformers are one of the most important machine learning workloads today. Training one is a very compute-intensive task, often taking days or weeks, and significant attention ha…
Deep Learning for Post-Processing Ensemble Weather Forecasts
Peter Grönquist, Chengyuan Yao, Tal Ben-Nun +4
Quantifying uncertainty in weather forecasts is critical, especially for predicting extreme weather events. This is typically accomplished with ensemble prediction systems, which c…
Predicting Weather Uncertainty with Deep Convnets
Peter Grönquist, Tal Ben-Nun, Nikoli Dryden +4
Modern weather forecast models perform uncertainty quantification using ensemble prediction systems, which collect nonparametric statistics based on multiple perturbed simulations.…
Asynchronous Decentralized SGD with Quantized and Local Updates
Giorgi Nadiradze, Amirmojtaba Sabour, Peter Davies +2
Decentralized optimization is emerging as a viable alternative for scalable distributed machine learning, but also introduces new challenges in terms of synchronization costs. To t…
Using Known Information to Accelerate HyperParameters Optimization Based on SMBO
Cheng Daning, Zhang Hanping, Xia Fen +2
Automl is the key technology for machine learning problem. Current state of art hyperparameter optimization methods are based on traditional black-box optimization methods like SMB…
Asynch-SGBDT: Asynchronous Parallel Stochastic Gradient Boosting Decision Tree based on Parameters Server
Cheng Daning, Xia Fen, Li Shigang +1
In AI research and industry, machine learning is the most widely used tool. One of the most important machine learning algorithms is Gradient Boosting Decision Tree, i.e. GBDT whos…
Weighted parallel SGD for distributed unbalanced-workload training system
Cheng Daning, Li Shigang, Zhang Yunquan
Stochastic gradient descent (SGD) is a popular stochastic optimization method in machine learning. Traditional parallel SGD algorithms, e.g., SimuParallel SGD, often require all no…