Website live & crawler blows a fuse

This website went live on 10th July but the usually well behaved crawler decided now was a good time to throw a wobbly.

We've decided to make some fairly drastic changes (that we'd had planned anyway) so we're a little behind schedule but we're going to roll with the punches.

Small sites crawl just fine but larger sites (100k pages+) are just bombing because of the page counts. There was a part of the crawler app that didn't use web services (we were trying to save time...) and that's the bit that's spat its dummy out. The moral of the story is don't cut corners. I'm sure I've learned that lesson before...

Anyway, the new crawler should be live tomorrow so we'll probably invite a few friends, etc to sign-up and see how much the word gets spread before we do any more promotion.

We hope to get around 50 or so users in the beta program before we close it to new sign-ups and run for a while to see how it goes then open it up again.

Any users that are signed up already may notice that the reporting interface has a few minor issues in IE7 - they should be sorted tomorrow.

Please remember to log any feedback, problems, etc via the support centre.