http://web.nexor.co.uk/mak/doc/robots/robots.html
World Wide Web Robots, Wanderers, and Spiders are all names for programs that traverse the Web automatically. This page is devoted to learning as much about these Spiders, their uses and what pitfalls to avoid.