works on
ai assistants 1empirical study 1robots.txt compliance 1search-augmented AI 1web crawling 1

From the 1 of 1 linked paper with an AI index.

collaborators

1 paper

cs.CY2026

Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers

Gabriel Lopez-Fonseca, David Rodriguez, Stefan Bechtold +1

The paper conducts an empirical study of ten generative AI assistants with web‑search capabilities to see whether they obey robots.txt restrictions, revealing varied compliance and…