Robots Search Engines Searching

Web robots (also known as crawlers or spiders) are programs that traverse the Web automatically, and which are used by search engines to index the Web, or part of it.

• Top/Computers/Internet/Web Design and Development/Authoring/Online Tools/robots.txt
• Top/Computers/Internet/E-mail/Spam/Preventing
• Top/Computers/Internet/Searching/Search Engines/Sitemaps
• Top/Computers/Software/Internet/Site Management/Log Analysis

• Robotstxt.org - Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.
  
• Robot IP Address - Brian Dunnintg provides a list of all the major search engine robot IP addresses, by full class C only.
  
• HTTP User Agent Index - An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.
  
• Search Engine Robots and Other User Agents - John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.
  
• User Agent String - Tool from ASAP Consulting s.r.o. for detailed user agent string analysis using an online form. Includes databases of browsers and robots.
  
• Bots vs Browsers - This large database lists user agents in categories and distinguishes between robots and browsers.
  
• List of Robot Agent Strings - A list from PGTS of Web robots with the identifying data they leave in Web site logs.
  
• All About Search Indexing Robots and Spiders - Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.
  
• User-Agents.org - Large list of search engine spiders, similar web robots, and Web browsers: their web-log identification and links to their originators.
  
• Search Engine IP Addresses - Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.
  
• ACAP - Automated Content Access Protocol - Standard being developed on behalf of content publishers to communicate permissions information more extensively than is the case with robots.txt. Project documents, implementation and background information.
  
  

Copyright © 1995-2009 Internet Advertising Solutions, Inc.
Copyright Notice | Privacy Policy | Site Map | APR