2.2k citations · 6.4k across the 57 of their papers we have counts for
7 papers · 2 filters
Is Feature Diversity Necessary in Neural Network Initialization?
Yaniv Blumenfeld, Dar Gilboa, Daniel Soudry
Standard practice in training neural networks involves initializing the weights in an independent fashion. The results of recent work suggest that feature "diversity" at initializa…
The Knowledge Within: Methods for Data-Free Model Compression
Matan Haroush, Itay Hubara, Elad Hoffer +1
Recently, an extensive amount of research has been focused on compressing and accelerating Deep Neural Networks (DNN). So far, high compression rate algorithms require part of the…
A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case
Greg Ongie, Rebecca Willett, Daniel Soudry +1
A key element of understanding the efficacy of overparameterized neural networks is characterizing how they represent functions as the number of weights in the network approaches i…
At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?
Niv Giladi, Mor Shpigel Nacson, Elad Hoffer +1
Background: Recent developments have made it possible to accelerate neural networks training significantly using large batch sizes and data parallelism. Training in an asynchronous…
Kernel and Rich Regimes in Overparametrized Models
Blake Woodworth, Suriya Gunasekar, Pedro Savarese +5
A recent line of work studies overparametrized neural networks in the "kernel regime," i.e. when the network behaves during training as a kernelized linear predictor, and thus trai…
How do infinite width bounded norm networks look in function space?
Pedro Savarese, Itay Evron, Daniel Soudry +1
We consider the question of what functions can be captured by ReLU networks with an unbounded number of units (infinite width), but where the overall network Euclidean norm (sum of…