Huge spike in crawl errors today - mozbot ignoring noindex tag?
-
Hi Mozzers,
Today I received a ton of errors and warnings in my weekly crawl due to the mozbot crawling my noindex'd search results pages, such as this - http://www.consumerbase.com/Mailing-Lists.html?q=Construction&type=bus&channel=all&page=7&order=title&orderBy=DESC See image: http://screencast.com/t/qaZzq78j2Udx
Anyone else seen a similar error this week with their crawl?
Thanks!
-
Great to hear!
-
After the last crawl, everything returned to normal and all the errors disappeared. Roger must have just had a hiccup.
-
Don't forget that Rogerbot is a crawler and not an Index. Google will crawl those pages too but will (generally) follow your instructions to noindex those pages.
Here is a little information on Rogerbot http://moz.com/help/pro/rogerbot-crawler
And here is a little information on the frequently asked questions for th diagnostics report http://moz.com/help/pro/crawl-diagnostics
Hope this helps.
Darin.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Site Crawl Stalled and Can't Restart
In my GreenSeed campaign, the site crawl continues to say "in progress." I can't figure out how to stop it or how to restart the site crawl. Can you please help?
Moz Pro | | Winger1 -
Increase in authorization permission errors (Access Denied - Error 403)
Hi MOZ community, Since last week when I changed my theme in a WP installation I noticed (in WMT and MOZ tool) that I have increased number in authorization permission errors (error 403-forbidden). What happens is that I received a 403 error for almost every single URL of my site. All these URLs are not "real" ones but they all have my email in the end. i.e. I get an 403 error for the "/contact/support@fantasylogic.com" whilst the real URL is just "/contact/" This happens, as I said, for almost every single page of my site. I have no other crawling or indexation issues, all URLs are correctly indexed. All new pages are correctly indexed as well. URIs ending with "support@fantasylogic.com" are not indexed off course. WP and all installed plugins & theme are on the latest available release. For SEO purposes I use Yoast SEO WP plugin. The site in questions is: fantasylogic.com Any suggestions would be highly appreciated. Thank you in advance
Moz Pro | | gpapatheodorou0 -
Seomoz crawl: 4XX (Client Error) How to find were the error are?
I got eight 404 errors with the Seomoz crawl, but the report does not says where the 404 page is linked from (like it does for dup content), or I'm I missing something? Thanks
Moz Pro | | PaddyDisplays0 -
Has any on else experienced a spike in crawl errors?
Hi, Since the last time our sites were crawled in SEOmoz they are all showing a spike in Errors. (Mainly duplicate page titles and duplicate content). We haven't changed anything to the structure of the sites but they are all using the same content management system. The image is an example of what we are witnessing for all our sites based on the same system. Is anyone else experiencing anything similar? or does anyone know of any changes that SEOmoz has implemented which may be affecting this? Thanks in advance, Anthony. WzdQV WzdQV WzdQV.jpg WzdQV.jpg
Moz Pro | | BallyhooLtd1 -
How do i fix these errors?
Hi How do i fix these errors? (attached) I dont understand them and dont see any instructions to fixing them.. Regards Chris capturefpa.jpg
Moz Pro | | chrisleonard0 -
Should I be running my crawl on our www address or our non-www address?
I currently run our crawl on oursitename.com, but am wondering if it should be run on www.oursitename.com instead.
Moz Pro | | THMCC0 -
Why does the crawl report say I should have meta description and title tags in my xml files?
Just had my first crawl report today which has been very useful in finding missing and duplicated title tags and meta descriptions but it has flagged up the fact that my xml files are missing these. Surely non HTML documents shouldn't have them (or need them) so why are they showing up in the report?
Moz Pro | | PandyLegend0 -
How to get seomoz to re-crawl a site?
I had a lot of duplicate content issues and have fixed all the other warnings. I want to check the site again.
Moz Pro | | adamzski0