Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference

Gaoyuan Du, Anam Nawaz Khan, Rex Zhou +4

Greedy decoding from large language models is commonly treated as deterministic. We show it is not precision-invariant: the same model, prompt, and decoding algorithm produce diffe…