10 citations · 10 across the 4 of their papers we have counts for
9 papers · 1 filter
Normalized Architectures are Natively 4-Bit
Maxim Fishman, Brian Chmiel, Ron Banner +2
Training large language models at 4-bit precision is critical for efficiency. We show that nGPT, an architecture that constrains weights and hidden representations to the unit hype…
FP4 All the Way: Fully Quantized Training of LLMs
Brian Chmiel, Maxim Fishman, Ron Banner +1
We demonstrate, for the first time, fully quantized training (FQT) of large language models (LLMs) using predominantly 4-bit floating-point (FP4) precision for weights, activations…
EXAQ: Exponent Aware Quantization For LLMs Acceleration
Moran Shkolnik, Maxim Fishman, Brian Chmiel +3
Quantization has established itself as the primary approach for decreasing the computational and storage expenses associated with Large Language Models (LLMs) inference. The majori…
Scaling FP8 training to trillion-token LLMs
Maxim Fishman, Brian Chmiel, Ron Banner +1
We train, for the first time, large language models using FP8 precision on datasets up to 2 trillion tokens -- a 20-fold increase over previous limits. Through these extended train…
Bimodal Distributed Binarized Neural Networks
Tal Rozen, Moshe Kimhi, Brian Chmiel +2
Binary Neural Networks (BNNs) are an extremely promising method to reduce deep neural networks' complexity and power consumption massively. Binarization techniques, however, suffer…
Colored Noise Injection for Training Adversarially Robust Neural Networks
Evgenii Zheltonozhskii, Chaim Baskin, Yaniv Nemcovsky +3
Even though deep learning has shown unmatched performance on various tasks, neural networks have been shown to be vulnerable to small adversarial perturbations of the input that le…