Cascade: GPU-Accelerated Out-of-Core Suffix Array Construction for LLM Corpora

Jamshed Khan, Prashant Pandey

Suffix arrays over LLM (pre)training corpora underpin deduplication, contamination testing, n-gram statistics, and training-data attribution, and every new corpus or tokenizer need…