Cascade: GPU-Accelerated Out-of-Core Suffix Array Construction for LLM Corpora

Jamshed Khan, Prashant Pandey

Suffix arrays power deduplication, contamination testing, and training data attribution over the corpora used to train large language models, and must be rebuilt for every new corp…