STEP 4
Search Engine Spiders

Spiders (also called "bots") move from website to website and page to page by following links. The Google spider known as Googlebot does a semi-regular "crawl" into the interior pages of sites that are already in its database.

This crawl is used by Google to index many or all of the interior pages of your website as well as find new pages (and new sites) to index.

You can tell whether your site has been crawled by looking for the annotation "Googlebot" in your visitor logs (check with your web host for information on accessing your log files).

After your pages have been crawled, the new versions of those page(s) are placed in Google's cache for your site. Once your site has been indexed by Google, you can view the latest version of your pages in Google's cache like this:

Search for your website URL on Google. After the description, you will see the word "cache". Click on it and you will see the version of your web page that Google has cached last.

Google's search engine spider (Googlebot) is the best friend a webmaster can have. In order to get listed in Google quickly (and most other search engines as well), all you have to do is get a few other websites that are already in Google to place links to your home page on their site (usually on a links page).

Googlebot visits sites with a high Google PageRank on the home page (typically PR5 or higher) virtually every day. Lower ranked sites will get visits at longer intervals.

You want the spider to visit your site as often as possible in order to pick up new pages and any changes to your old ones. To this end, you need to boost your home page's Page-Rank to at least PR5 as quickly as possible. You can easily do this simply by getting more high-quality inbound links.

Changes to your home page also help attract Googlebot, so keep it as fresh as possible.

Note: Google has another spider with the notation "mediapartners". The mediapartners spider is used in conjunction with their AdSense content-matching advertising program, and it has nothing to do with search engine rankings.