Search Engine Indexing

Every search engine relies on a database of web sites in order to provide search results. When a search is performed, the search engine's software scans the database and retrieves relevant results. It is the database on which the search engine relies.

There are two fundamental ways of compiling a database: automated indexing or manual indexing.

Automated indexing involves a piece of software called a spider or robot which continually processes web sites by scanning them and indexing them according to content. The spider acts like an AI web surfer, reading over the content of pages to determine how they should be categorized. It will even follow links to other pages scanning those as well. With the vast number of web pages in existence, spiders perform a task that would exhaust a human.

Spiders are not perfect, however, as they can only judge a page according to its programming. Hence, there exist some human compiled directories which rely on manual submissions and updates by real people. These directories have the advantage that sites are more likely to be indexed correctly and the quality of sites tends to be higher. But because of the time involved in manual indexing, human-indexed directories tend to contain outdated information compared with spider-indexed directories.

<-- Previous   Next -->