There's some interesting sites out there Posted by Dan Frost on Wed, 25/07/2007 - 16:23
We've been watching sites go through the crawler and the first, most immediate issue is that our crawler isn't coping with malformed HTML as well as it might. There's a fix on the way and the crawler should be able to handle those issues shortly. We'll re-crawl and notify those users that had that problem. Unclosed tags and so on are commonplace and we had a basic HTML validator on the "to do" list but I think we'll do it sooner rather than later. If your site crawls but shows 0 or 1 page crawled it's either because you've got major issues with your HTML or none of the URLs on the site are the same as as the initial URL - eg: if you tell crawlscore that your URL is http://google.co.uk but your links are actually http://www.google.co.uk then crawlscore will treat them as external URLs. There's an argument either way for following sub-domain URLs but for the moment, we're treating each site individually. To crawl a subdomain simply add it as a site and it will be crawled independently. As far as we know, most search engines treat subdomain differently anyway so we'll do the same :) We're trying to communicate directly with users where possible to let them know of issues (either their end or ours) but we're getting a high number of sign-ups so please bear with us. |