When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings

Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt +2

We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of att…