Getting The Most From Search Engines

Evolution of search engine technology

Search engines originally indexed by a process of manually adding pages. They ranked sites using techniques such as how many times a keyword appeared on a page, or what information was contained in meta tags in the page heading. In only a few years, however, search technology has undergone tremendous change.

Search engines now use "spiders" to continually find and add new sites to their databases. Spiders are software programs that work their way through pages on Web sites by following links contained within. As the spider encounters a link, it checks to see if the linked page is in the search database and if not, adds it. Spiders can also check to see if the page is in the index but has been modified, and update the index.

The results displayed when you do a search at a site such as Google or MSNSearch are the product of matching your search terms to the content of pages in the search index (which will often produce tens of thousands of results for simple or unfocused searches), and then ranking and displaying those pages. Rankings are based on the concept of "importance" - that is, the more sites link to your site, and the more highly ranked those sites are- the better your own site's ranking will be.

Google, currently the most highly rated search engine, describes its ranking process ("PageRank") as follows:

PageRank performs an objective measurement of the importance of pages and is calculated by solving an equation of up to hundreds of millions of variables and billions of terms. PageRank uses the link structure of the network as an organizational tool. In essence, Google interprets a link from Page A to Page B as a "vote" by Page A for Page B. Google assesses a page's importance by the votes it receives. Google also analyzes the page that casts the vote. Votes cast by pages that are themselves "important" weigh more heavily and help to make other pages "important."