Design, implementation and experiment of a YeSQL Web Crawler

Pierre Joulin, Romain Deveaud, Eric SanJuan-Ibekwe +2

We describe a novel, "focusable", scalable, distributed web crawler based on GNU/Linux and PostgreSQL that we designed to be easily extendible and which we have released under a GN…