5 papers · 1 filter
Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?
Qingjie Zhang, Xingzhang Ren, Zixuan Chen +6
Pretraining corpus composition shapes LLM capabilities, but it often remains hidden even when model weights are released. Prior work has inferred corpus mixtures or traced specific…
Auditing Chinese Web-scale Corpora via Sampled BPE Token Statistics
Qingjie Zhang, Ziqi Tang, Jie Zhang +7
Chinese web pollution has surfaced in LLMs, motivating audits of upstream Chinese corpora. However, auditing such corpora faces three challenges: (1) their web-scale size makes ful…
ConsisGuard: Aligning Safety Deliberation with Policy Enforcement in LLM Guardrails
Yan Wang, Zhixuan Chu, Zihao Xue +9
Reasoning-based LLM guardrails improve safety moderation by generating explicit rationales before issuing final decisions. However, their rationales do not always lead to faithful…
How LoRA Remembers? A Parametric Memory Law for LLM Finetuning
Ziwen Xu, Haiwen Hong, Linsong Yu +4
Large Language Models (LLMs) must continuously learn and update knowledge to remain effective in dynamic real-world environments. While Low-Rank Adaptation (LoRA) is widely used fo…
AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset
Bingxiang He, Wenbin Zhang, Jiaxi Song +11
Preference learning is critical for aligning large language models (LLMs) with human values, yet its success hinges on high-quality datasets comprising three core components: Prefe…