Showing cs.CLShow all
3 papers · 1 filter
cs.CL2026
Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
Xu Guo, Runyu Peng, Jian Tong +4
Large language models (LLMs) rely on web-scale corpora for pre-training. The noise inherent in these datasets tends to obscure meaningful patterns and ultimately degrade model perf…
cs.CL2024
WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset
Jiantao Qiu, Haijun Lv, Zhenjiang Jin +23
This paper presents WanJuan-CC, a safe and high-quality open-sourced English webtext dataset derived from Common Crawl data. The study addresses the challenges of constructing larg…
cs.CL2024
Data-free Weight Compress and Denoise for Large Language Models
Runyu Peng, Yunhua Zhou, Qipeng Guo +4
Large Language Models (LLMs) are reshaping the research landscape in artificial intelligence, particularly as model parameters scale up significantly, unlocking remarkable capabili…