User-agent: * Disallow: / # NOTE This is mainly to prevent AI companies from scraping the entire website. If you are a researcher, historian or hobbyist, you are free to automate requests to the site so long as it's reasonable or somewhat limited or somewhat targeted