Huge spike in crawl errors today - mozbot ignoring noindex tag?
-
Hi Mozzers,
Today I received a ton of errors and warnings in my weekly crawl due to the mozbot crawling my noindex'd search results pages, such as this - http://www.consumerbase.com/Mailing-Lists.html?q=Construction&type=bus&channel=all&page=7&order=title&orderBy=DESC See image: http://screencast.com/t/qaZzq78j2Udx
Anyone else seen a similar error this week with their crawl?
Thanks!
-
Great to hear!
-
After the last crawl, everything returned to normal and all the errors disappeared. Roger must have just had a hiccup.
-
Don't forget that Rogerbot is a crawler and not an Index. Google will crawl those pages too but will (generally) follow your instructions to noindex those pages.
Here is a little information on Rogerbot http://moz.com/help/pro/rogerbot-crawler
And here is a little information on the frequently asked questions for th diagnostics report http://moz.com/help/pro/crawl-diagnostics
Hope this helps.
Darin.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Strange "?offset" URL found with content crawl issues
I recently recieved a slew of content crawl issues via Moz for URL's that I have never seen before For example:
Moz Pro | | HannahPalamara
Standard URL: https://skilldirector.com/news,
Newly identified URL: https://skilldirector.com/news?offset=1469542207800&category=Competency+Management). Does anyone know where the URL comes from and how to fix it?0 -
Pages with Duplicate Content Error
Hello, the result of renewed content appeared in the scan results in my Shopify Store. But these products are unique. Why am I getting this error? Can anyone please help to explain why? screenshot-analytics.moz.com-2021.10.28-19_53_09.png
Moz Pro | | gokimedia0 -
Question about Crawl Diagnostics - 4xx (Client Error) report
Hi here, I was wondering if there is a way to find out the originating page where a broken link is found from the 4xx (Client Error) report. I can't find a way to know that, and without that information is very difficult for me to fix any possible 404 related issues on my website. Any thoughts are very welcome! Thank you in advance.
Moz Pro | | fablau0 -
Crawl Diagnostics: Next crawl date is in the past
Hi - I have quite a few crawl diagnostic errors and warnings. I have attempted to fix many of them but noticed this note at the bottom of the crawl diagnostics chart: "Last Crawl Completed: Mar. 22nd, 2013 Next Crawl Starts: Mar. 29th, 2013" It looks like SEOMoz thinks the next crawl date is Mar 29th, 2013, which is two weeks ago. Is there any way to "force" the crawl and get it back on regular schedule? This may have happened when my account was disabled because my credit card expired...Thoughts?
Moz Pro | | 6thirty0 -
How to Fix the Errors with Duplicate Title or Content?
The latest Crawl Diagnostic has found 160 Errors on my site.
Moz Pro | | hanmark
And my error is, that the same content or title is used on two different! pages:
on both my root domain (han-mark.com) and the www subdomain. What does it matter (with or without www)? How serious is that error? Do I need to fix all the errors (and hundreds of warnings too)? What's the best practice? Is there any Guide on how to do it
or Tools for doing it the fast way? Viggo Joergensen0 -
Crawl Diagnostics shows two title and meta tag errors but they are false positives.
I got one hit each on "Missing Meta Description Tag" and "Title Missing or Empty" but in the source of my page they are clearly there: <title>Protein Powder | Compare and Get the Best Prices</title> <meta name="keywords" content="protein powder, whey protein, protein supplement, whey protein isolate, hydrolyzed whey" /> I understand there are conventions which may or may not be followed by Drupal (I read an earlier question where ordering and W3C conventions were suggested) but i'm not sure how to fix them given Drupal will just overwrite any hand editing the next time something is built and importantly, I can't get the crawl to work on cue - it works on the automatic once a week crawl in the main campaign summary but every time I've specifically used the Crawl Test tool it gives me a "There was an error submitting your request to the crawler. Please try again later" so I can't really test any changes. Given Google seems to be recognising the title tag - ie showing it in the results - Do I put this down as seomoz just not working? Kind Regards, Brian
Moz Pro | | btrr690 -
Some questions on Canonical tag AND 301 redirect
Hi everyone, I'm new here - always loved SEOMoz and glad to be part of the Pro community now. I have 2 questions regarding the Canonical URL tag. Some background info: We used to run an OsCommerce store, and recently migrated to Magento. In doing so, we right away created 301 redirects of the old category pages (OsCommerce) to the new category pages (Magento) via the Magento admin. Example: www.example.com/old-widget-category.html
Moz Pro | | yacpro13
301 redicrected to
www.example.com/new-widget-category.html In Magento admin, we have enabled the Canonical tag for all product and category pages. Here's how Magento sets up the Canonical tag: The URL of interest which we want to rank is:
www.example.com/new-widget-category.html However Magento sets up the canonical tag on this page to point to:
www.example.com/old-widget-category.html When using the SEOMoz On Page Report Card, it pick this up as an error because the Canonical tag is pointing to a different URL. However, if we dig a little deeper, we see that the URL being pointed to
www.example.com/old-widget-category.html
has a 301 redirect to
www.example.com/new-widget-category.html
which is the URL we wan to rank. So because we set up a 301 redirect of the old-page to the new-page, on the new-page the canonical tag points to the old-page. Question 1)
What are you opinions on this? Do you think this method of setting up the Canonical tag is acceptable? Second question... We use pagination for category pages, so if we have 50 products in one category, we would have 5 pages of 10 products. The URL's would be: www.example.com/new-widget-category.html (which is the SAME as ?p=1)
www.example.com/new-widget-category.html?p=1
www.example.com/new-widget-category.html?p=2
www.example.com/new-widget-category.html?p=3
www.example.com/new-widget-category.html?p=4
www.example.com/new-widget-category.html?p=5 Now ALL the URLs above have the canonical tag set as:
<link rel="canonical" href="http://www.example.com/new-widget-category" /> However, the content of each page (page 1, 2, 3, 4, 5) is different because different products are displayed. So far most what I read regarding the Canonical tag is that it is used for pages that have the same content but different URLs. I would hope that Google would combine the content of all 5 pages and view the result as a single URL www.example.com/new-widget-category Question 2) Is using the canonical tag appropriate in the case described above? Thanks !0
