This article is intended as a primer for those interested in how search engines and directories index web sites. It is not a substitute for professional assistance, since the search engine optimization industry is one of the most dynamic on the Internet.
Definitions
While the term "search engine" tends to be a catch-all phrase for any web site that indexes and lists other sites, there are two distinct forms:
Directory - managed by human editor(s) who review sites that are submitted and those they find while surfing the web. They will add sites that they deem would add value to their directory. Directories are often broken down by category, and ranking is determined by the web site's name, description and category as it appears in the directory. As a rule, it is tougher to get listed in a directory than in a search engine. Examples of directories are Yahoo, Open Directory Project and LookSmart.
Search Engine - uses a software "robot" or "spider" to navigate the web and find sites to add to its index, generally without human intervention. Sites are ranked based on algorithms known only to the search engine developers, however some assumptions can be made by those familiar with the workings of search engines. Examples of search engines are Go Network (formerly InfoSeek), AltaVista and Excite.
Note: many of the directories and search engines have merged somewhat to expand their listings. For example, a search in Yahoo lists the Yahoo directory results first, and then Inktomi search engine results. Similarly, a search on Lycos will often have Open Directory Project results mixed in.
How Web Sites are Indexed
Directories
Since Yahoo is by far the most popular directory on the web we'll use it as an example. Although it is theoretically possible for a Yahoo editor to find your site and add it to their directory without being notified of its existence, it is extremely unlikely. The best way to get listed is by submitting your web site. Great care must be taken when submitting a site to Yahoo, since once listed it is extremely difficult if not impossible to get a listing changed.
When submitting your web site you will be required to input the web site name, category (sometimes more than one) and site description, along with the usual contact information. The web site name is usually your company name and must be kept short. Sites are listed alphabetically within a category, so if you can opt for a name closer to the beginning of the alphabet it is advisable to do so. However, do not try to get a site listed as "AAA Widget Shop" when your name is actually "Widget Shop". Doing so will only raise the ire of the editor and may result in your site getting listed poorly - or not at all.
To choose a category, perform several searches on keyword phrases that your site visitors would most likely use. Make a note of which categories appear first on these searches. Also note which categories your competitors appear in. These are generally the best categories to try to get listed in.
Now comes the hard part: writing the site description. Directories do not determine relevancy from the title or meta tags on your web site, so the description you submit to a directory can be different from that on your site. Yahoo states that there is a limit 25 words in the site description, though in practice it may be somewhat less. Your site description should have plenty of carefully chosen keywords in it, and contain no hyperbole such as "best widgets on the web" or "we're #1". For a company that sells candle making supplies a good description might read "offering a line of candles and candlemaking supplies including wax, scents, wicks, pottery, and glassware." Too long a description may get edited or dropped altogether. It is best stay well within their stated limits in order to get your site listed.
Once your site is submitted, it can take anywhere from days to months to obtain a listing. Yahoo currently accepts only 25 percent of sites submitted. A well-designed site with informative content is much more likely to get listed than a one page site touting the latest multi-level marketing scheme. Even then there is no guarantee that your site will be listed with the categories and description you submitted, or even listed at all.
Search Engines
A search engine is a completely different animal altogether. In most cases you submit your site to a search engine simply by typing in your URL and email address. Once submitted, the search engine "spider" will visit your site the next time it makes its rounds. Sometimes it will index the main page only and come back later for the other pages. Other times it will navigate through the entire site at once. Each search engine has it's own technique. Getting your site listed can take anywhere from days to weeks.
Ranking is determined by the structure and content of the web page - the HTML tags (title, meta and alt tags) and the text of the page itself. Since most search engines can index many pages on your site - not just the home page - it is important to ensure that each page has been optimized for the search engines. Each search engine uses a different technique to determine ranking of web sites; these algorithms are highly protected and not made publicly available.
Search Engine Optimization (SEO) is an industry that is rapidly growing in part due to the huge number of sites on the web competing against each other. It is getting harder to stand out from the crowd and many businesses are using SEO specialists to assist them with their search engine rankings. An SEO specialist constantly monitors the search engines watching for changes and adjusts their optimization strategies to accommodate these changes.
While Search Engine Optimization is far to complex to discuss in detail here, some basic pointers can be given:
- Make sure each page on your site has well developed Title, Meta Description, Meta Keywords and Alt tags focused on the content of that particular page. The Title tag is what you see at the very top of your monitor screen when viewing a web site (in windows this is usually a blue bar). This is also what often appears as the hyperlink in a search engine listing. The Meta Description and Meta Keywords tags are only viewable in the web page source code but are very important in determining ranking. Alt tags are the text seen when a page is viewed with image loading turned off in the user's web browser, or the text that pops up when the mouse is moved over an image. A common mistake is to include every possible keyword on every single page - you should instead include only on those keywords relevant to that page and avoid broad keywords like "business" or "computers". If a keyword phrase is important enough to be included you should have at least one page on your site devoted to that topic.
The text of the web page itself is very important in determining ranking of a web site. It is not enough to have your keywords in the Title, Meta and Alt tags mentioned above. They also should appear within the page content several times, preferably in a heading tag such as <H1> Important Text Here</H1>. Search engines cannot read text that is part of a graphic, so make sure any important keywords are repeated as actual text. Some search engines require a minimum number of words to appear as text on a page before being indexed. A minimum of 75 words should suffice; most web pages have far more.
Be careful of search engine spamming (or "spamdexing") - overuse of keyword repetition or the placing of "hidden" words such as white text on a white background. This can result in your web site being banner from the search engine.
Search engines also tend to have a difficult time navigating sites that use frames. Use of the NoFrames tag will help alleviate this to some degree by providing text and links that are visible to search engines.
Larger web sites consisting of many page often obtain better rankings than smaller web sites. The more pages that a web site consists of the more exposure in the search engines it will receive, and the more likely it will be found on a particular search.
There is a technique called gateway (also called doorway) pages that can be used in the event a web site's design does not lend itself to optimization or in a highly competitive industry. Gateway pages consist of mostly text and are optimized for a single keyword phrase. They generally contain numerous links back to the main web site. Since the gateway page is designed for search engine indexing and not aesthetics, there is a better chance of that page receiving a higher ranking. When a person performs a search that lists a gateway page, they will visit the gateway page first before clicking on to the main site.
Link popularity is also becoming a factor in site ranking among some search engines. The idea is that the more valuable and informative your web site content is, the more people will link to you from their sites.
Dynamically generated web pages, such as those often found on shopping cart product pages, cannot be indexed by search engines. If you see a "?" symbol in the URL, that indicates such a web page. Pages with content included in them via a server side include (SSI) can be indexed however.
Summary Points
Search Engine Optimization is intrinsic to site design and should be taken into consideration before beginning the construction of your web site.
On the web "Content is King". A professionally designed web site with informative content that has been optimized will generally do well in both directories and search engines, and is more likely to have inbound links from other sites as a result.
Larger sites do better than smaller sites as a rule.
Since SEO is a rapidly changing area, there are no guarantees of ranking or inclusion of a page or pages in a search engine or directory. However, certain techniques can be used that have had good results in the past.
