Continual Pretraining on Encrypted Synthetic Data for Privacy-Preserving LLMs

Honghao Liu, Xuhui Jiang, Chengjin Xu +4

Preserving privacy in sensitive data while pretraining large language models on small, domain-specific corpora presents a significant challenge. In this work, we take an explorator…