On the Structure of Floating-Point Noise in Batch-Invariant GPU Matrix Multiplication

Tadisetty Sai Yashwanth

Floating-point non-associativity makes fundamental deep learning operations, such as matrix multiplication (matmul) on GPUs, inherently non-deterministic. Despite this, the statist…