I got a 404 in the Crawl Test Tool Report
-
I, yesterday i ran an crawl on http://www.everlastinggarden.nl and i get an 404. Does anybody know why this happens?
<colgroup><col width="1535"></colgroup>
| # ---------------------------------------- |
| Crawl Test Tool Report | Moz,http://pro.seomoz.org/tools/crawl-test |
| www.everlastinggarden.nl |
| Report created: 15 Jul 18:34 |
| # ---------------------------------------- |
| URL,Time Crawled,Title Tag,Meta Description,HTTP Status Code,Referrer,Link Count,Content-Type Header,4XX (Client Error),5XX (Server Error),Title Missing or Empty,Duplicate Page Content,URLs with Duplicate Page Content (up to 5),Duplicate Page Title,URLs with Duplicate Title Tags (up to 5),Long URL,Overly-Dynamic URL,301 (Permanent Redirect),302 (Temporary Redirect),301/302 Target,Meta Refresh,Meta Refresh Target,Title Element Too Short,Title Element Too Long,Too Many On-Page Links,Missing Meta Description Tag,Search Engine blocked by robots.txt,Meta-robots Nofollow,Blocked by X-robots,X-Robots-Tag Header,Blocked by meta-robots,Meta Robots Tag,Rel Canonical,Rel-Canonical Target,Blocking All User Agents,Blocking Google,Blocking Yahoo,Blocking Bing,Internal Links,Linking Root Domains,External Links,Page Authority |
| http://www.everlastinggarden.nl,2014,404 : Received 404 (Not Found) error response for page.,Error attempting to request page |Best regards,
Jos
-
Erin,
thank you very much for your effort. I already contacted my host on this problem, but they say is that they are only blocking Yandex and that every other crawler is allowed.
In the past my client did have similair errors (website not reachable) but that was solved i understood. I will contact you via in private for additional information.
Also I will contact the host again and ask them to investigate the matter.
Best regards, Jos
-
Hi Jos,
Thanks for reaching out! My name is Erin, and I'm on the Moz Help Team. Our crawls have been delayed, and we're currently working through the backlog. Thanks for your patience! Keep in mind that the Crawl Test is cached for 48 hours, so if you run them too close together you'll see the same data.
I curled your site from our servers, and I came back with a 404. Usually this means that your site is blocking us on the server level- and explains why your site comes back with a 200 from a Header Checker, but a 404 from our servers. I did a little digging but wasn't able to come up with much, so you'll want to loop in your web dev to see if they can shed some more light on what might be happening. I'm sorry there's not more I can do for you on the support side with this one!
I don't want to share any private site details over the interwebz, so if you have any follow up questions email the Help Team.
I hope that helps, and happy Monday!
Cheers,
Erin
-
Hi! I've asked our help team to come and take a look at this and investigate. Thanks for your patience!
-
Can anybody help? I re-issued another crawl, but this one is already 4 days on it's way ????
See attachement.
Best regards, Jos
-
Er_Maqui, Keri,
thank you for you responses. I informed with the host, the only block Yandex everything else is allowed.
I did a re-do of the crawl with the same result ;-(
@Keri, it's a 404 on the homepage.... :
http://www.everlastinggarden.nl,2014,404 : Received 404 (Not Found) error response for page.,Error attempting to request page.
Also in my campaign it gives a 404. If i check the a-record, all looks good: https://www.whatsmydns.net/#A/www.everlastinggarden.nl
Best regsrds, Jos
-
Hi! If you download the CSV of your crawl report, it will list the referring URL that is pointing to the page with a 404.
-
Try to re-do the crawl test. This can be a puntual problem.
For getting more info, check your server log. Try to enter the page before re-do the test for verify all it's working.
Also, if you can, check if your hosting is blocking some user-agents (question it to them) for reduce traffic.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl report shows that it gets 4xx errors for pages that work fine. Why?
On the crawl report it has all these "Critical Crawler Issues". They all say "4xx Error", yet when i click on the link from the crawler report, it goes to a perfectly functioning page, not a 404 page or anything. If i click in it actually says it's a 403 error. It's all for pages generated by the IDX solution for our real estate website. Is Moz broken or am i missing something? Here are a couple examples: <dl class="crawl-page-details-list"> <dd class="crawl-page-details-list-emphasis">https://teamvivi.com/homes-for-sale-map-search/</dd> <dd class="crawl-page-details-list-emphasis"> <dl class="crawl-page-details-list"> <dd class="crawl-page-details-list-emphasis">https://teamvivi.com/email-alerts/</dd> </dl> </dd> </dl>
Moz Bar | | TeamViviRealEstate0 -
How non-US Moz customers will use Keyword Explorer after the Keyword Difficulty tool is retired?
The new Moz Keyword Explorer looks good but its search volume is US based and completely useless for non-US websites. This is from Rand's post: "while the tool can search any Google domain in any country, the volume numbers will always be for US-volume. In the future, we hope to add volume data for other geos as well." In the Keyword Difficulty tool, Moz shows Google search volume data, which is similar to what I see in the Google Keyword Planner and Google Search Console. For example, keyword X in the Australian search market has 6-7k searches in the Google Keyword Planner and 8k searches in Moz. The very same keyword has 118k-300k search volume in the new Keyword Explorer! Obviously this new search volume is not useful in the Australian market. I often used the Keyword Difficulty tool to identify new keyword opportunities but what can I do to complete the same tasks after they retire the tool?
Moz Bar | | Gyorgy.B2 -
Moz Tool Bar Annoyance - How do I make the green keyword difficulty box go away?
A highly annoying light green keyword difficulty box just started showing up to the right of the search box. How do I make this go away? ulBzP9W
Moz Bar | | JenKeller0 -
Accessing old reports
Hi there, Is there any way of accessing reports for a certain campaign that are a year old? To my knowledge the reports had not been generated, and if they were, they have not been saved in the custom reports section.
Moz Bar | | WebhouseISP
Is there anyway of generating a report from months ago? Thanks a million!0 -
Have there been recent changes to the On-Page Grader tool?
I use this page all the time to grade my primary and secondary keywords and for four years now have consistently managed to achieve A-C grades for all. Today however, I noticed I am flagging F grades for 50% of the keywords. I do make a note of the initial grades for all pages (as the majority of the keywords I grade using the tool are NOT added/tracked in MOZ 'campaigns'). To make sure I wasn't going mad I went back and graded some keywords I had achieved A-C grades for just 6 days ago. The changes are as follows: keyword 1 initially graded an A - 6 days later is still an A keyword 2 initially graded C and is now graded as an F keyword 3 initially graded C and now an F keyword 4 initially a C grade and now an F keyword 5 initially a B grade and has remained a B. Out of interest I then went and added one of the keywords that is now graded an F (but last week was a C) into the 'Add and manage Page Grades' section and graded once again - it's now grading this as a C. I then removed this keyword from my tracked keywords and graded again using the 'On-Page Grader' tool and ... it's back down to an 'F'. Please could you shed some light on what's going on? I don't want to track each keyword I grade over time but I do want the parameters for grading to remain the same if I use the 'On-Page Grader' tool - what's changed? I haven't received any alerts about any changes to the tool so if anyone could shed some light on any changes I would appreciate it.
Moz Bar | | em_h_uk0 -
How to get Universal Results to Show in Reports?
This has been a source of frustration for me for a long time. I have asked many times and never found anything in the support documentation on how to add this. Is it something that is simply not offered? If so is it not outrageous to others that this simple feature is not present? What I want: I would like to be able to include Universal Results icons in my rank reports. Why? Well as is the case much of the time my clients Google profile is included in the maps results (thanks to our optimization services) but they are not on the first page. The client can't see the results of this type of SEO. It shows up when logged in but not in the exported reports. How easy would it be to include these? It seems like a no-brainer. If anyone knows where to add them or why they don't automatically include them please let me know. If this feature is not offered I call upon my fellow SEO's to demand this feature. If anyone knows of a way to contact the developers to recommend adding this feature I would also appreciate that information. Thank you Universal Result: Local
Moz Bar | | TotalMarketExposure
You're in it!
Position: before 1st natural result
Contains: 7 results0 -
Is there a good local competitive analysis tool for the UK
Hello Mozers I have learnt from the forum about the competitive analysis tool http://www.51blocks.com/online-marketing-tools/free-local-analysis/ and followed the link - only to find that it is for the US. I'm in the UK and need to access this for google.uk - does anyone know of a UK equivalent? Could I also check with the forum to ask the question that got me into this in the first place. It's probably a daft question. For the Ranking report in Moz, do I just need to put the local city name in front of the keyword to get the local rank. I have been doing this and assuming it's ok, and my local ranking does show up. Thanks Catherine
Moz Bar | | catherine-2793880 -
Is there any tool that checks the rankings at any time I want?
My problem is that in Moz when I check ranking it shows me only on a weekly basis. Acording to MOZ I was on rank 11 yesterday. This is not what my SEO guy has sent me in his report. And usually he is right. Would be nice to have a ranking report for each day. What is the sense of a ranking tool if I don't know what happened between 2 ranking dates? How can I check that? Is there any rank history tool? Or a tool that will check in real time. The way I do it sometimes is to go on google.com/ncr to see the us version of google because I am in Europe. Rank tracker does not work for me at the moment. Get ranking button is inactive. My old keyword list has not been imported from SEOMOZ
Moz Bar | | Valentinopolus0