Trajectory Normalized Gradients for Distributed Optimization
arXiv:1901.08227
Abstract
Recently, researchers proposed various low-precision gradient compression, for efficient communication in large-scale distributed optimization. Based on these work, we try to reduce the communication complexity from a new direction. We pursue an ideal bijective mapping between two spaces of gradient distribution, so that the mapped gradient carries greater information entropy after the compression. In our setting, all servers should share a reference gradient in advance, and they communicate via the normalized gradients, which are the subtraction or quotient, between current gradients and the reference. To obtain a reference vector that yields a stronger signal-to-noise ratio, dynamically in each iteration, we extract and fuse information from the past trajectory in hindsight, and search for an optimal reference for compression. We name this to be the trajectory-based normalized gradients (TNG). It bridges the research from different societies, like coding, optimization, systems, and learning. It is easy to implement and can universally combine with existing algorithms. Our experiments on benchmarking hard non-convex functions, convex problems like logistic regression demonstrate that TNG is more compression-efficient for communication of distributed optimization of general functions.
References in corpus (13)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Communication Efficient Distributed Optimization using an Approximate Newton-type Method
- Sparsified SGD with Memory
- Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
- Better Mini-Batch Algorithms via Accelerated Gradient Methods
- Communication/Computation Tradeoffs in Consensus-Based Distributed Optimization
- An Alternative View: When Does SGD Escape Local Minima?
- Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization
- Communication-Efficient Algorithms for Decentralized and Stochastic Optimization
- Improved Optimization of Finite Sums with Minibatch Stochastic Variance Reduced Proximal Iterations