Decode Late, Multiply Once: Rethinking the Dot Product for Quantized Weights

Jonathan Tapson

Large neural networks are increasingly deployed with 1-4 bit weights and 16- or 32-bit activations, yet their dot products still use one multiplier per element. We show that a leng…