Investigating Data Contamination for Pre-training Language Models

Minhao Jiang, Ken Ziyu Liu, Ming Zhong +4

Language models pre-trained on web-scale corpora demonstrate impressive capabilities on diverse downstream tasks. However, there is increasing concern whether such capabilities mig…