A Scalable Crawling Algorithm Utilizing Noisy Change-Indicating Signals

Róbert Busa-Fekete, Julian Zimmert, András György +6

Web refresh crawling is the problem of keeping a cache of web pages fresh, that is, having the most recent copy available when a page is requested, given a limited bandwidth availa…