72 citations · 542 across the 65 of their papers we have counts for
6 papers · 2 filters
Optimal Mini-Batch Size Selection for Fast Gradient Descent
Michael P. Perrone, Haidar Khan, Changhoan Kim +3
This paper presents a methodology for selecting the mini-batch size that minimizes Stochastic Gradient Descent (SGD) learning time for single and multiple learner problems. By deco…
Negative sampling in semi-supervised learning
John Chen, Vatsal Shah, Anastasios Kyrillidis
We introduce Negative Sampling in Semi-Supervised Learning (NS3L), a simple, fast, easy to tune algorithm for semi-supervised learning (SSL). NS3L is motivated by the success of ne…
Demon: Improved Neural Network Training with Momentum Decay
John Chen, Cameron Wolfe, Zhao Li +1
Momentum is a widely used technique for gradient-based optimizers in deep learning. In this paper, we propose a decaying momentum (\textsc{Demon}) rule. We conduct the first large-…
Distributed Learning of Deep Neural Networks using Independent Subnet Training
Binhang Yuan, Cameron R. Wolfe, Chen Dun +3
Distributed machine learning (ML) can bring more computational resources to bear than single-machine learning, thus enabling reductions in training time. Distributed learning parti…
MLSys: The New Frontier of Machine Learning Systems
Alexander Ratner, Dan Alistarh, Gustavo Alonso +66
Machine learning (ML) techniques are enjoying rapidly increasing adoption. However, designing and implementing the systems that support ML models in real-world deployments remains…
Compressing Gradient Optimizers via Count-Sketches
Ryan Spring, Anastasios Kyrillidis, Vijai Mohan +1
Many popular first-order optimization methods (e.g., Momentum, AdaGrad, Adam) accelerate the convergence rate of deep learning models. However, these algorithms require auxiliary p…