Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality

Vedant Badoni, Danqi Chen, Xinyi Wang

The performance of modern language models depends critically on pretraining data composition. Yet existing data selection methods rely on auxiliary classifiers for document scoring…