Change Rate Estimation and Optimal Freshness in Web Page Crawling

Konstantin Avrachenkov, Kishor Patil, Gugan Thoppe

For providing quick and accurate results, a search engine maintains a local snapshot of the entire web. And, to keep this local cache fresh, it employs a crawler for tracking chang…