Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web

Nicolas Steinacker-Olsztyn, Devashish Gosain, Ha Dao

Large Language Models (LLMs) are increasingly relying on web crawling to stay up to date and accurately answer user queries. These crawlers are expected to honor robots.txt files,…