Underscoring the Problem: Why Softpick Fails at Initialization

Aryan Sood, Jaikaran Singh, Ishaan Bansal

Softmax attention gives every token a nonzero weight, which in trained models concentrates into attention sinks and massive activations that widen the dynamic range low-precision i…