2 citations · 5 across the 7 of their papers we have counts for
3 papers · 1 filter
The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning
Nikhil Ghosh, Spencer Frei, Wooseok Ha +1
In this work, we investigate the dynamics of stochastic gradient descent (SGD) when training a single-neuron autoencoder with linear or ReLU activation on orthogonal data. We show…
The Power of External Memory in Increasing Predictive Model Capacity
Cenk Baykal, Dylan J Cutler, Nishanth Dikkala +3
One way of introducing sparsity into deep networks is by attaching an external table of parameters that is sparsely looked up at different layers of the network. By storing the bul…
Alternating Updates for Efficient Transformers
Cenk Baykal, Dylan Cutler, Nishanth Dikkala +3
It has been well established that increasing scale in deep transformer networks leads to improved quality and performance. However, this increase in scale often comes with prohibit…