Robots Search Engines Searching

Robots

Web robots (also known as crawlers or spiders) are programs that traverse the Web automatically, and which are used by search engines to index the Web, or part of it.

• Top/Computers/Software/Internet/Site Management/Log Analysis
• Top/Computers/Internet/Abuse/Spam/Preventing
• Top/Computers/Internet/Web Design and Development/Authoring/Online Tools/robots.txt

• - Andreas Staeding's large list of search engine spiders, similar Web robots, and Web browsers: their web-log identification and links to their originators.
  
• - Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.
  
• - José Luis Pellicer's searchable database of robots, spiders and other user agents for programs that surf the web.
  
• - A list from PGTS of Web robots with the identifying data they leave in Web site logs.
  
• - Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.
  
• - John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.
  
• - An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.
  
  

Copyright © 1995-2008 Internet Advertising Solutions, Inc.
Copyright Notice | Privacy Policy | Site Map | APR