Auditing Chinese Web-scale Corpora via Sampled BPE Token Statistics

Qingjie Zhang, Ziqi Tang, Jie Zhang +7

Chinese web pollution has surfaced in LLMs, motivating audits of upstream Chinese corpora. However, auditing such corpora faces three challenges: (1) their web-scale size makes ful…