4 citations · 8 across the 13 of their papers we have counts for
12 papers · 1 filter
UE5M3 FP4 Block Scaling for Stable Language Model Pretraining
Robert Hu, Carlo Luschi, Paul Balanca
Stable 4-bit floating-point (FP4) pretraining is difficult because the E2M1 payload represents only a narrow range of magnitudes. NVIDIA's Transformer Engine \nv{} recipe addresses…
MXNorm: Reusing MXFP block scales for efficient tensor normalisation
Callum McLean, Luke Y. Prince, Alexandre Payot +2
Matrix multiplication performance has long been the major bottleneck to scaling deep learning workloads, which has stimulated the design of new accelerators that use increasingly l…
Ground-Truth Subgraphs for Better Training and Evaluation of Knowledge Graph Augmented LLMs
Alberto Cattaneo, Carlo Luschi, Daniel Justus
Retrieval of information from graph-structured knowledge bases represents a promising direction for improving the factuality of LLMs. While various solutions have been proposed, a…
Elucidating the Design Space of FP4 training
Robert Hu, Carlo Luschi, Paul Balanca
The increasing computational demands of foundation models have spurred research into low-precision training, with 4-bit floating-point (\texttt{FP4}) formats emerging as a frontier…
Optimal Formats for Weight Quantisation
Douglas Orr, Luka Ribar, Carlo Luschi
Weight quantisation is an essential technique for enabling efficient training and deployment of modern deep learning models. However, the recipe book of quantisation formats is lar…
The Role of Graph Topology in the Performance of Biomedical Knowledge Graph Completion Models
Alberto Cattaneo, Stephen Bonner, Thomas Martynec +4
Knowledge Graph Completion has been increasingly adopted as a useful method for helping address several tasks in biomedical research, such as drug repurposing or drug-target identi…
Scalify: scale propagation for efficient low-precision LLM training
Paul Balança, Sam Hosegood, Carlo Luschi +1
Low-precision formats such as float8 have been introduced in machine learning accelerated hardware to improve computational efficiency for large language models training and infere…
u-P: The Unit-Scaled Maximal Update Parametrization
Charlie Blake, Constantin Eichenberg, Josef Dean +7
The Maximal Update Parametrization (P) aims to make the optimal hyperparameters (HPs) of a model independent of its size, allowing them to be swept using a cheap proxy model rat…
Reducing the Cost of Quantum Chemical Data By Backpropagating Through Density Functional Theory
Alexander Mathiasen, Hatem Helal, Paul Balanca +6
Density Functional Theory (DFT) accurately predicts the quantum chemical properties of molecules, but scales as . Schütt et al. (2019) successfully appro…
SparQ Attention: Bandwidth-Efficient LLM Inference
Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley +3
The computational difficulties of large language model (LLM) inference remain a significant obstacle to their widespread deployment. The need for many applications to support long…
Generating QM1B with PySCF
Alexander Mathiasen, Hatem Helal, Kerstin Klaser +6
The emergence of foundation models in Computer Vision and Natural Language Processing have resulted in immense progress on downstream tasks. This progress was enabled by datasets w…
Training and inference of large language models using 8-bit floating point
Sergio P. Perez, Yan Zhang, James Briggs +6
FP8 formats are gaining popularity to boost the computational efficiency for training and inference of large deep learning models. Their main challenge is that a careful choice of…