I got a 404 in the Crawl Test Tool Report

[IMforYou]

I, yesterday i ran an crawl on http://www.everlastinggarden.nl and i get an 404. Does anybody know why this happens?

<colgroup><col width="1535"></colgroup>
| # ---------------------------------------- |
| Crawl Test Tool Report | Moz,http://pro.seomoz.org/tools/crawl-test |
| www.everlastinggarden.nl |
| Report created: 15 Jul 18:34 |
| # ---------------------------------------- |
| URL,Time Crawled,Title Tag,Meta Description,HTTP Status Code,Referrer,Link Count,Content-Type Header,4XX (Client Error),5XX (Server Error),Title Missing or Empty,Duplicate Page Content,URLs with Duplicate Page Content (up to 5),Duplicate Page Title,URLs with Duplicate Title Tags (up to 5),Long URL,Overly-Dynamic URL,301 (Permanent Redirect),302 (Temporary Redirect),301/302 Target,Meta Refresh,Meta Refresh Target,Title Element Too Short,Title Element Too Long,Too Many On-Page Links,Missing Meta Description Tag,Search Engine blocked by robots.txt,Meta-robots Nofollow,Blocked by X-robots,X-Robots-Tag Header,Blocked by meta-robots,Meta Robots Tag,Rel Canonical,Rel-Canonical Target,Blocking All User Agents,Blocking Google,Blocking Yahoo,Blocking Bing,Internal Links,Linking Root Domains,External Links,Page Authority |
| http://www.everlastinggarden.nl,2014,404 : Received 404 (Not Found) error response for page.,Error attempting to request page |

Best regards,

Jos

[IMforYou]

Erin,

thank you very much for your effort. I already contacted my host on this problem, but they say is that they are only blocking Yandex and that every other crawler is allowed.

In the past my client did have similair errors (website not reachable) but that was solved i understood. I will contact you via in private for additional information.

Also I will contact the host again and ask them to investigate the matter.

Best regards, Jos

[ErinMcCaul]

Hi Jos,

Thanks for reaching out! My name is Erin, and I'm on the Moz Help Team. Our crawls have been delayed, and we're currently working through the backlog. Thanks for your patience! Keep in mind that the Crawl Test is cached for 48 hours, so if you run them too close together you'll see the same data.

I curled your site from our servers, and I came back with a 404. Usually this means that your site is blocking us on the server level- and explains why your site comes back with a 200 from a Header Checker, but a 404 from our servers. I did a little digging but wasn't able to come up with much, so you'll want to loop in your web dev to see if they can shed some more light on what might be happening. I'm sorry there's not more I can do for you on the support side with this one!

I don't want to share any private site details over the interwebz, so if you have any follow up questions email the Help Team.

I hope that helps, and happy Monday!

Cheers,

Erin

[KeriMorgret]

Hi! I've asked our help team to come and take a look at this and investigate. Thanks for your patience!

[IMforYou]

Can anybody help? I re-issued another crawl, but this one is already 4 days on it's way ????

See attachement.

Best regards, Jos

crawlreport.jpg

[IMforYou]

Er_Maqui, Keri,

thank you for you responses. I informed with the host, the only block Yandex everything else is allowed.

I did a re-do of the crawl with the same result ;-(

@Keri, it's a 404 on the homepage.... :

http://www.everlastinggarden.nl,2014,404 : Received 404 (Not Found) error response for page.,Error attempting to request page.

Also in my campaign it gives a 404. If i check the a-record, all looks good: https://www.whatsmydns.net/#A/www.everlastinggarden.nl

Best regsrds, Jos

[KeriMorgret]

Hi! If you download the CSV of your crawl report, it will list the referring URL that is pointing to the page with a 404.

[Er_Maqui]

Try to re-do the crawl test. This can be a puntual problem.

For getting more info, check your server log. Try to enter the page before re-do the test for verify all it's working.

Also, if you can, check if your hosting is blocking some user-agents (question it to them) for reduce traffic.