50 citations · 79 across the 7 of their papers we have counts for
7 papers
Tensor-Parallelism with Partially Synchronized Activations
Itay Lamprecht, Asaf Karnieli, Yair Hanani +2
Training and inference of Large Language Models (LLMs) with tensor-parallelism requires substantial communication to synchronize activations. Our findings suggest that with a few m…
DropCompute: simple and more robust distributed synchronous training via compute variance reduction
Niv Giladi, Shahar Gottlieb, Moran Shkolnik +5
Background: Distributed training is essential for large scale training of deep neural networks (DNNs). The dominant methods for large scale DNN training are synchronous (e.g. All-R…
Flood forecasting with machine learning models in an operational framework
Sella Nevo, Efrat Morin, Adi Gerzi Rosenthal +28
The operational flood forecasting system by Google was developed to provide accurate real-time flood warnings to agencies and the public, with a focus on riverine floods in large,…
Physics-Aware Downsampling with Deep Learning for Scalable Flood Modeling
Niv Giladi, Zvika Ben-Haim, Sella Nevo +2
Background: Floods are the most common natural disaster in the world, affecting the lives of hundreds of millions. Flood forecasting is therefore a vitally important endeavor, typi…
At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?
Niv Giladi, Mor Shpigel Nacson, Elad Hoffer +1
Background: Recent developments have made it possible to accelerate neural networks training significantly using large batch sizes and data parallelism. Training in an asynchronous…
Evaluating and Calibrating Uncertainty Prediction in Regression Tasks
Dan Levi, Liran Gispan, Niv Giladi +1
Predicting not only the target but also an accurate measure of uncertainty is important for many machine learning applications and in particular safety-critical ones. In this work…
Augment your batch: better training with larger batches
Elad Hoffer, Tal Ben-Nun, Itay Hubara +3
Large-batch SGD is important for scaling training of deep neural networks. However, without fine-tuning hyperparameter schedules, the generalization of the model may be hampered. W…