home *** CD-ROM | disk | FTP | other *** search
/ bioinformatics.org / bioinformatics.org_software.tar / www.bioinformatics.org / robots.txt
Text File  |  2013-01-20  |  369b  |  22 lines
  1. # @(#) standard robots.txt file
  2.  
  3. User-agent: *
  4. Crawl-delay: 3
  5. Disallow: /admin/
  6. Disallow: /images/
  7. Disallow: /includes/
  8. Disallow: /manual/
  9. Disallow: /websvn/
  10. Disallow: /cgi-bin/cvsweb.cgi/
  11.  
  12. User-agent: googlebot
  13. Crawl-delay: 3
  14. Disallow: /admin/
  15. Disallow: /images/
  16. Disallow: /includes/
  17. Disallow: /manual/
  18. Disallow: /websvn/
  19. Disallow: /cgi-bin/cvsweb.cgi/
  20.  
  21. # end of file
  22.