A question about Mozbot and a recent crawl on our website.
-
Hi All,
Rogerbot has been reporting errors on our website's for over a year now, and we correct the issues as soon as they are reported.
However I have 2 questions regarding the recent crawl report we got on the 8th.
1.) Pages with a "no-index" tag are being crawled by roger and are being reported as duplicate page content errors. I can ignore these as google doesnt see these pages, but surely roger should ignore pages with "no-index" instructions as well? Also, these errors wont go away in our campaign until Roger ignores the URL's.
2.) What bugs me most is that resource pages that have been around for about 6 months have only just been reported as being duplicate content. Our weekly crawls have never picked up these resources pages as being a problem, why now all of a sudden? (Makes me wonder how extensive each crawl is?)
Anyone else had a similar problem?
Regards
GREG
-
Its pretty big
Over 1000 Pages in the index, and many more internal URLs to crawl that have a no-index tag. (booking forms etc)
Ill see if we can archive our other campaigns and let roger crawl our main site properly.
-
How big is your website Greg ?
-
Thanks Nakul,
I do a weekly scan with Xenu which doesn't have a URL limit like SF.
I was under the impression a full scan of the site was done each week, but as you say, its being scanned in chunks, divided across our 3 other websites.
If this is the case, it would be great to let Mozbot know were to crawl to avoid unnecessary resources being used up when it could be scanning our most important pages.
Greg
-
Greg The crawl is limited to 10,000 (Total) for all your 5 campaigns. As far as whether or not Roger-Bot should ignore Noindex - Here's what I think - I think the intent of that tool here is to find issue. In this scenario, Roger bot is making sure you are aware of the fact that some of those pages have a noindex. Roger does not know whether it's intentional or not. You can also do a deeper crawl and do a deep dive into your website by using Screaming Frog SEO Spider http://www.screamingfrog.co.uk/seo-spider/ It does a great job of doing a deep crawl when you want it since it's a desktop software and you can set all sorts of options and identify issues.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz crawl duplicate pages issues
Hi According to the moz crawl on my website I have in the region of 800 pages which are considered internal duplicates. I'm a little puzzled by this, even more so as some of the pages it lists as being duplicate of another are not. For example, the moz crawler considers page B to be a duplicate of page A in the urls below: Not sure on the live link policy so ive put a space in the urls to 'unlive' them. Page A http:// nuchic.co.uk/index.php/jeans/straight-jeans.html?manufacturer=3751 Page B http:// nuchic.co.uk/index.php/catalog/category/view/s/accessories/id/92/?cat=97&manufacturer=3603 One is a filter page for Curvety Jeans and the other a filter page for Charles Clinkard Accessories. The page titles are different, the page content is different so Ive no idea why these would be considered duplicate. Thin maybe, but not duplicate. Like wise, pages B and C are considered a duplicate of page A in the following Page A http:// nuchic.co.uk/index.php/bags.html?dir=desc&manufacturer=4050&order=price Page B http:// nuchic.co.uk/index.php/catalog/category/view/s/purses/id/98/?manufacturer=4001 Page C http:// nuchic.co.uk/index.php/coats/waistcoats.html?manufacturer=4053 Again, these are product filter pages which the crawler would have found using the site filtering system, but, again, I cannot find what makes pages B and C a duplicate of A. Page A is a filtered result for Great Plains Bags (filtered from the general bags collection). Page B is the filtered results for Chic Look Purses from the Purses section and Page C is the filtered results for Apricot Waistcoats from the Waistcoat section. I'm keen to fix the duplicate content errors on the site before it goes properly live at the end of this month - that's why anyone kind enough to check the links will see a few design issues with the site - however in order to fix the problem I first need to work out what it is and I can't in this case. Can anyone else see how these pages could be considered a duplicate of each other please? Checking ive not gone mad!! Thanks, Carl
Moz Pro | | daedriccarl0 -
Is there a tool to crawl website meta tags to locate any in an incorrect language?
Example: I'm interested in crawling a regional site for Brazil to find any meta tags that are still in English with the goal being to fix any localization issues.
Moz Pro | | mattsolar0 -
Website data updates
Hello All, I have downloaded some reports for "following" links from SEO Moz and no doubt the report is very comprehensive. However, I know some links listed in the reports does have rel="nofollow" tag attached to it but it is list still shown that they are following links. Can some please guide be to understand why this has happened. Is it possible that SEO Moz has old data of our site because these changes were recently done (2-3 weeks before)..? OR Have I misunderstood following as dofollow..? Any help would be highly appreciated. Thanks, moosa.
Moz Pro | | moosa0 -
Crawl diagnostic Notices for rel Canonical increased
Hello, We just signed up for SEO Moz, and are reviewing the results of our second web crawl. Our Errors and Warnings summary have been reduced, but our Notices for Rel Canonical have skyrocketed from 300 to over 5,500. We are using a WP with the Headway theme and our pages already have the rel=canonical along wiht rel=author. Any ideas why this number would go up so much in one week? Thank you, Michael
Moz Pro | | MKaloud0 -
Crawl Diagnostic Errors
Hi there, Seeing a large number of errors in the SEOMOZ Pro crawl results. The 404 errors are for pages that look like this: http://www.example.com/2010/07/blogpost/http:%2F%2Fwww.example.com%2F2010%2F07%2Fblogpost%2F I know that t%2F represents the two slashes, but I'm not sure why these addresses are being crawled. The site is a wordpress site. Anyone seen anything like this?
Moz Pro | | rosstaylor0 -
A suggestion to help with linkscape crawling and data processing
Since you guys are understandably struggling with crawling and processing the sheer number of URLs and links, I came up with this idea: In a similar way to how SETI@Home (is that still a thing? Google says yes: http://setiathome.ssl.berkeley.edu/) works, could SEOmoz use distributed computing amongst SEO moz users to help with the data processing? Would people be happy to offer up their idle processor time and (optionally) internet connections to get more accurate, broader data? Are there enough users of the data to make distributed computing worthwhile? Perhaps those who crunched the most data each month could receive moz points or a free month of Pro. I have submitted this as a suggestion here:
Moz Pro | | seanmccauley
http://seomoz.zendesk.com/entries/20458998-crowd-source-linkscape-data-processing-and-crawling-in-a-similar-way-to-seti-home1 -
How do I find questions with new responses?
I'm really enjoying hanging out here so far. I was wondering if there is an easy way to find out which of the discussions that I am taking part in have new responses in them. I know I get an email whenever a question I have subscribed to gets an answer, but is there a way to see this when I am on the website? I see that I can go to "My questions" and "Questions I have asked" or "Questions I have answered" but on that screen it doesn't tell me which one(s) of those questions have new answers. Am I missing something?
Moz Pro | | MarieHaynes1 -
How do I delete a question?
Having problems with this question, old edits get saved as copies of the question, even this one had to be edited twice, originally an old version of an edited question. How do I DELETE a question I authored (like this one) I see no button for it.
Moz Pro | | AspenFasteners0