36 citations · 124 across the 26 of their papers we have counts for
13 papers · 1 filter
TORQUE: Optimizing What (not) to Quantize Before and After Rotation
Ran Ben Basat, Michael Mitzenmacher, Shay Vargaftik
Uniform random rotations are an effective preprocessing step for quantization: they make normalized coordinate distributions approximately Gaussian, enabling the use of codebooks o…
Entropy-Constrained Adaptive Stochastic Quantization
Ran Ben Basat, Yaniv Ben-Itzhak, Michael Mitzenmacher +1
Adaptive stochastic quantization (ASQ) is a recently introduced quantization approach that optimizes the Mean Squared Error (MSE) for a given input while preserving unbiasedness. I…
Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven
Ran Ben-Basat, William Kuszmaul, Michael Mitzenmacher +2
Uniform random rotations (URRs) are a common preprocessing step in modern quantization approaches used for gradient compression, inference acceleration, KV-cache compression, model…
A Note on TurboQuant and the Earlier DRIVE/EDEN Line of Work
Ran Ben-Basat, Yaniv Ben-Itzhak, Gal Mendelson +3
This note clarifies the relationship between the recent TurboQuant work and the earlier DRIVE (NeurIPS 2021) and EDEN (ICML 2022) schemes. DRIVE is a 1-bit quantizer that EDEN exte…
DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce
Wenchen Han, Shay Vargaftik, Michael Mitzenmacher +1
Multi-hop all-reduce is the de facto backbone of large model training. As the training scale increases, the network often becomes a bottleneck, motivating the reduction of the volu…
Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression
Wenchen Han, Shay Vargaftik, Michael Mitzenmacher +2
Gradient aggregation has long been identified as a major bottleneck in today's large-scale distributed machine learning training systems. One promising solution to mitigate such bo…
Optimal and Near-Optimal Adaptive Vector Quantization
Ran Ben-Basat, Yaniv Ben-Itzhak, Michael Mitzenmacher +1
Quantization is a fundamental optimization for many machine-learning use cases, including compressing gradients, model weights and activations, and datasets. The most accurate form…
THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression
Minghao Li, Ran Ben Basat, Shay Vargaftik +4
Deep neural networks (DNNs) are the de facto standard for essential use cases, such as image classification, computer vision, and natural language processing. As DNNs and datasets…
DoCoFL: Downlink Compression for Cross-Device Federated Learning
Ron Dorfman, Shay Vargaftik, Yaniv Ben-Itzhak +1
Many compression techniques have been proposed to reduce the communication overhead of Federated Learning training procedures. However, these are typically designed for compressing…
QUIC-FL: Quick Unbiased Compression for Federated Learning
Ran Ben Basat, Shay Vargaftik, Amit Portnoy +3
Distributed Mean Estimation (DME), in which clients communicate vectors to a parameter server that estimates their average, is a fundamental building block in communication-eff…
EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learning
Shay Vargaftik, Ran Ben Basat, Amit Portnoy +3
Distributed Mean Estimation (DME) is a central building block in federated learning, where clients send local gradients to a parameter server for averaging and updating the model.…
DRIVE: One-bit Distributed Mean Estimation
Shay Vargaftik, Ran Ben Basat, Amit Portnoy +3
We consider the problem where clients transmit -dimensional real-valued vectors using bits each, in a manner that allows the receiver to approximately reconstruc…
RADE: Resource-Efficient Supervised Anomaly Detection Using Decision Tree-Based Ensemble Methods
Shay Vargaftik, Isaac Keslassy, Ariel Orda +1
Decision-tree-based ensemble classification methods (DTEMs) are a prevalent tool for supervised anomaly detection. However, due to the continued growth of datasets, DTEMs result in…