1 citations · 1 across the 1 of their papers we have counts for
3 papers · 1 filter
Adaptive Braking for Mitigating Gradient Delay
Abhinav Venigalla, Atli Kosson, Vitaliy Chiley +1
Neural network training is commonly accelerated by using multiple synchronized workers to compute gradient updates in parallel. Asynchronous methods remove synchronization overhead…
Pipelined Backpropagation at Scale: Training Large Models without Batches
Atli Kosson, Vitaliy Chiley, Abhinav Venigalla +2
New hardware can substantially increase the speed and efficiency of deep neural network training. To guide the development of future hardware architectures, it is pertinent to expl…
Online Normalization for Training Neural Networks
Vitaliy Chiley, Ilya Sharapov, Atli Kosson +5
Online Normalization is a new technique for normalizing the hidden activations of a neural network. Like Batch Normalization, it normalizes the sample dimension. While Online Norma…