4 citations · 4 across the 8 of their papers we have counts for
8 papers · 1 filter
UE5M3 FP4 Block Scaling for Stable Language Model Pretraining
Robert Hu, Carlo Luschi, Paul Balanca
Stable 4-bit floating-point (FP4) pretraining is difficult because the E2M1 payload represents only a narrow range of magnitudes. NVIDIA's Transformer Engine \nv{} recipe addresses…
A Practical Investigation of Training-free Relaxed Speculative Decoding
Guoxuan Xia, Luka Ribar, Paul Balanca
Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard spec…
MXNorm: Reusing MXFP block scales for efficient tensor normalisation
Callum McLean, Luke Y. Prince, Alexandre Payot +2
Matrix multiplication performance has long been the major bottleneck to scaling deep learning workloads, which has stimulated the design of new accelerators that use increasingly l…
Elucidating the Design Space of FP4 training
Robert Hu, Carlo Luschi, Paul Balanca
The increasing computational demands of foundation models have spurred research into low-precision training, with 4-bit floating-point (\texttt{FP4}) formats emerging as a frontier…
Scalify: scale propagation for efficient low-precision LLM training
Paul Balança, Sam Hosegood, Carlo Luschi +1
Low-precision formats such as float8 have been introduced in machine learning accelerated hardware to improve computational efficiency for large language models training and infere…
Reducing the Cost of Quantum Chemical Data By Backpropagating Through Density Functional Theory
Alexander Mathiasen, Hatem Helal, Paul Balanca +6
Density Functional Theory (DFT) accurately predicts the quantum chemical properties of molecules, but scales as . Schütt et al. (2019) successfully appro…
Generating QM1B with PySCF
Alexander Mathiasen, Hatem Helal, Kerstin Klaser +6
The emergence of foundation models in Computer Vision and Natural Language Processing have resulted in immense progress on downstream tasks. This progress was enabled by datasets w…
Training and inference of large language models using 8-bit floating point
Sergio P. Perez, Yan Zhang, James Briggs +6
FP8 formats are gaining popularity to boost the computational efficiency for training and inference of large deep learning models. Their main challenge is that a careful choice of…