Crawl test from tools

[hanzoz]

Hi,

I notice that the crawl test which is from the Research Tools doesn't really get a new crawl even though there is 2 crawl per day. It will only provide the data which was acquire from the crawl diagnostics in my pro account. There is no point for me to get the data which I get from my crawl diagnostic isn't it? Even seomoz provided with more than 2 crawl per day also useless in this case.

This whole thing doesn't make sense as the crawl diagnostics will only perform a full crawl test once every week. but even the crawl test also not helping any thing out for me.

[hanzoz]

Everyone, thanks for helping... Greatly appreciated.

[Peterli]

Hi there Hanzo,

Thanks for reaching out!  it looks like you are having some trouble with the crawl test.  Keri is correct that our crawl test uses "Cached" information to construct your crawl, which can take up to 2 days to clear.  While I am not sure why the sites are still reporting 404s, there's a few things that could explain why you might be getting a stale crawl:

• Your server simply did not update the changes you've made on the backend.
• There might an issue with our crawler, which in my last status there isn't a outage that I know of.

I looked into your crawl data and the 404s seems to be confirmed by the report, if you download your latest report and do a sort you should see that

Hope I was able to provide some insight, please let me know if you have questions!

Best,

[KeriMorgret]

I believe the crawl test has a cache of a couple of days. I'm asking someone from the help desk to come in and confirm, but it may be Monday before they have a chance to do so.

[hanzoz]

I know there is third party app which I can use to crawl,

but then isn't the crawl test is there for us to do testing instead of using other app rather than seomoz's tool?

[Chris.Menke]

You may want to give screaming frog or xenu's link sleuth.  They should be able to crawl you site  and give you the info you need.

[hanzoz]

From what I do with the crawl test, it doesn't show me the latest crawl. I need to check whether all the error 404 or related error was being solved. It only show me the old data which was crawl before but not the latest one.

[Chris.Menke]

Hanzo, what is it that you need to do that the crawl test isn't helping you with?