MOZ crawler 404 errors on wordpress
-
Hi all,
I've got hundreds of issues coming up on the MOZ crawler with 404 errors, I don't know what these URL's are. Here's a couple of examples;
http://www.theswagbagco.co.uk/category/watford/http%3A%2F%2Fwww.theswagbagco.co.uk%2F2015%2F10%2F15%2Fnew-products-2%2F
http://www.theswagbagco.co.uk/2015/10/01/thank-you-epsom/http%3A%2F%2Fwww.theswagbagco.co.uk%2F2015%2F10%2F01%2Fthank-you-epsom%2FSee the first one is one page with a different url appended, the second is the same thank-you-epsom url. How would I find out where these are even being linked from?
-
Hi there! Tawny from the Help Team here. Thanks for writing in - this is a great question.
In order to find the page our crawler was on when it encountered a 404 error, I'd recommend requesting the full crawl CSV via the 'Email CSV' button in your 'Site Crawl' section.
From there, you'll want to check the referrer URL for the page that 404s. To do that, find the URL of the page in column A of the CSV and locate the referrer URL for that page in column AM. The URL in column AM is the page our crawler was on when it ran into the link for the page that results in a 404 in the browser.
I hope this helps identify where these are coming from - if there's any additional information you're needing, please don't hesitate to let us know! You can write in to help@moz.com any time if you need a bit more guidance.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page Optimization Error
Hello.
Moz Bar | | csgosmurfcart
When I try to use 'On Page Grader' on specific site, I get an error message. "
Page Optimization Error
There was a problem loading this page. Please make sure the page is loading properly and that our user-agent, rogerbot, is not blocked from accessing this page.
"
example : https://www.csgosmurfkart.com Site's robots.txt settings are good. and I think there's no blocking factor. But On Page Grader cannot crawl the sites.
But campaign crawler is working well on the site. only On Page Grader is not working.. What should I change my server's setting or site's setting for crawling site on my site?
I'm using wordpress on google cloud Thank you.0 -
Cannot Crawl ... 612 : Page banned by error response for robots.txt.
I tried to crawl www.cartronix.com and I get this error: 612 : Page banned by error response for robots.txt. I have a robots.txt file and it does not appear to be blocking anything www.cartronix.com/robots.txt Also, Search Console is showing "allowed" in the robots.txt test... I've crawled many of our other sites that are similarly set up without issue. What could the problem be?
Moz Bar | | 1sixty80 -
Error 406 with crawler test
hi to all. I have a big problem with the crawler of moz on this website: www.edilflagiello.it. On july with the old version i have no problem and the crawler give me a csv report with all the url but after we changed the new magento theme and restyled the old version, each time i use the crawler, i receive a csv file with this error: "error 406" Can you help me to understan wich is the problem? I already have disabled .htacces and robots.txt but nothing. Website is working well, i have used also screaming frog as well.
Moz Bar | | ArchieDonnithorne1 -
Moz crawler only crawls one page?!
Hello there, I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled. We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working. Hopefully someone can point us at the right direction. Thanks in advance, Jeremy
Moz Bar | | mediaxplain.nl0 -
MOZ crawler has been finding a lot of 803 and 804 errors
During last 3 weeks MOZ crawler has been finding a lot of 803 and 804 errors. Meanwhile all pages seem to be working fine. What could cause it?
Moz Bar | | Paruyr0 -
Moz bar chrome
Hi
Moz Bar | | Emanuele_Ricci
moz bar does not work on chrome, it works correctly on firefox.
The bar is there but not showing any data...
Any idea? It can be something that is blocking moz bar on chrome browser or what else????
Thanks1 -
Crwal errors : duplicate content even with canonical links
Hi I am getting some errors for duplicate content errors in my crawl report for some of our products www.....com/brand/productname1.html www.....com/section/productname1.html www.....com/productname1.html we have canonical in the header for all three pages <link rel="canonical" href="www....com productname1.html"=""></link rel="canonical" href="www....com>
Moz Bar | | phes0 -
Moz crawl suddenly shows much less pages from what I really have
Hi! Moz crawl suddenly shows much less pages from what I really have and from what they used to show after completing the crawl. Should I be worried? What could that be? Regards, Yossey
Moz Bar | | Joseph-Green-SEO1