167 citations · 181 across the 12 of their papers we have counts for
3 papers · 1 filter
Attention Is All You Need For Mixture-of-Depths Routing
Advait Gadhikar, Souptik Kumar Majumdar, Niclas Popp +3
Advancements in deep learning are driven by training models with increasingly larger numbers of parameters, which in turn heightens the computational demands. To address this issue…
Cyclic Sparse Training: Is it Enough?
Advait Gadhikar, Sree Harsha Nelaturu, Rebekka Burkholz
The success of iterative pruning methods in achieving state-of-the-art sparse networks has largely been attributed to improved mask identification and an implicit regularization in…
Masks, Signs, And Learning Rate Rewinding
Advait Gadhikar, Rebekka Burkholz
Learning Rate Rewinding (LRR) has been established as a strong variant of Iterative Magnitude Pruning (IMP) to find lottery tickets in deep overparameterized neural networks. While…