Understanding Silent Data Corruption in LLM Training

Jeffrey Ma, Hengzhi Pei, Leonard Lausen +1

As the scale of training large language models (LLMs) increases, one emergent failure is silent data corruption (SDC), where hardware produces incorrect computations without explic…