Dan Frost's blog

Every week or so I'll be posting "celebrity site crawls" and talk about the results.

So this week in celebrity site crawls it's the battle of the giants.

In the corner with the bean bags and M&Ms dispenser, hailing from Mountain View, doing no evil whatsoever but still weighing in with a market capitalization of $137b, is the King of spin, the geek with a mean streak, Gooooooogle.

This post is now here.

Most people in SEO have heard the term "semantic search" - this means that a search engine is for a specific niche with the aim being that you get more relevant results for your searches.

Let's say you search on Google for "Derby" - you could mean the Kentucky Derby, Derby hat, Derby in England, hotels in Derby and so on.

I've been mulling over this article for some time now so I could draw some sensible conclusions from it but I've decided against doing it because there are far too many variables in there.

I recently stumbled upon a very interesting research project about how Googlebot, Slurp and MSNbot crawled pages on a very large site (billions of pages).

Googlebot activity

I've read many articles on this subject but being as we've actually written a web crawler I think we're pretty well placed to give good advice on how a website should be built to be easily crawled :)

1. Well structured HTML

The blog post I had in mind for this has been very, very well covered in Wim Leers post on improving Drupals page loading performance.

What I will do is blog about the specific issues I found with crawlscore.com and how I addressed them.

Because crawlscore.com has been through a couple of re-designs and also because I've made some mistakes when putting the site together, it needed some maintenance to ensure we make the most of each crawlers visit.

We're pondering on creating an open, collaborative study into how often web crawlers visit websites.

The idea is that with minor PHP/Apache mods you can turn robots.txt into an executable PHP file and basically submit the fact it has been requested to a central resource.

We've had some feedback from customers asking for a version of Crawl Score to crawl their 10-30 page websites saying that our starter package with 100 page limit is too much and it got us thinking.