Moz "Crawl Diagnostics" doesn't respect robots.txt
-
Hello, I've just had a new website crawled by the Moz bot. It's come back with thousands of errors saying things like:
- Duplicate content
- Overly dynamic URLs
- Duplicate Page Titles
The duplicate content & URLs it's found are all blocked in the robots.txt so why am I seeing these errors?
Here's an example of some of the robots.txt that blocks things like dynamic URLs and directories (which Moz bot ignored):Disallow: /?mode=
Disallow: /?limit=
Disallow: /?dir=
Disallow: /?p=*&
Disallow: /?SID=
Disallow: /reviews/
Disallow: /home/Many thanks for any info on this issue.
-
Hi Si, has this issue been resolved?
-
Hey Si,
Thanks for writing in. It doesn't seem that we are having an overarching issue with our crawler ignoring robots.txt files so I did some research in Google Webmaster Tools and it looks like most crawlers require an asterisk in the disallow directive to recognize that all pages of a dynamic URL are being disallowed. If you look in the "Pattern Matching" section of this resource here: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449, that should give you more information about setting up the robots.txt with the correct disallow directives to block those pages.
If you add in the astrisk to the disallow directive and you are still seeing these pages crawled, it would help if you sent in an email with your campaign information to our support desk at help@moz.com so we can have our engineers look into this more directly.
I hope this helps.
Chiaryn
-
If you have an "index,(no)follow" meta on those pages I think they will be crawled even though you have them blocked in robots.txt. So by adding "noindex" on those pages it might work as you want it to.
-
Is the / actually in the URL at that spot? Or is your link like http://www.example.com/abcd?p=147
If you give an example full URL that includes one of your blocked dynamic URLs we can take a better look. If your robots is setup correctly, it shouldn't find that stuff but give us more info if you're able.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Did Moz Bar change to have no Keyword capabilities?
I'm trying to use it during SEO training over here and the KW button goes to a "Get Page Optimization Score with MozBar Premium - Try Free", with a link that takes you to: https://hsinfo.moz.com/mozpro/mozbar/lander?utm_medium=cpc&utm_source=google&utm_campaign=Brand | NA&utm_adgroup=Brand - MozBar&utm_term=mozbar&gclid=CjwKCAjwxOCRBhA8EiwA0X8hi4Tx1YaOVwzWZaCGmzmYdBO4JEON8YlRMw52stp2AyfEBbH4uWDnARoCum0QAvD_BwE? Didn't this used to accept keywords and allow keyword checking? My training materials have the KW button behaving like this: [1] Do Keyword Research in MozBar.
Moz Bar | | EricaJorgensen
1. Click the icon with KW and a magnifying glass.
2. Enter a term related to your subject. For example, "cyber security".
There is a section telling you the keyword score, the relevancy to your page, and giving you optimizations that you can make to the page regarding this term.... Thus, I feel sure KW had useful and free features and not a button for a trial and paid Moz Bar Premium account. What is the pricing for this feature now? Or am I missing something? Thanks,
Tracy!0 -
Why is Moz Crawling More Pages Than My Site Actually Has?
Hi I have a site that only has 5k pages but Moz has crawled 50K pages on the site when I initiated the site crawl. I don't exactly know why Moz is reporting me back so many pages but I was wondering why this is and if any of you out in the Moz community know anything about this. Thanks
Moz Bar | | drewstorys0 -
Moz Keyword Tool Monthly Volume
Ive recently put together a Keyword List of about 100 keywords on the Moz Keyword Explorer tool. One keyword, aerial filming, stood out as very low search volume of 51 - 100. I took the same 100 keywords and passed them through the Google Keyword Planner by Google AdWords. Aerial Filming has an average search volume of 1k - 10k according to the Keyword Planner. Even though Keyword Planner gives me a range of 1k - 10k, the lowest number is still 10 times higher than what the Moz Keyword Explorer was indicating. This drastic difference of volume was consistent across all 100 keywords. All of the Monthly Volume numbers were divided by 10. Why does Moz Keyword Tool display a search volume that is 10x less than what Google Keyword Planner is suggesting?
Moz Bar | | fictionarts0 -
Does Moz's keyword tool pull data from your IP address?
Does anyone know how Moz's keyword tool pulls their keyword ranks? Do they take it based off of the IP (history and cookies) that is being used? I am trying to find a way to collect keyword data that is neutral and not based off of my previous searches, etc. TIA
Moz Bar | | ReviveMedia0 -
Why isn't the Moz bar data populating for Yahoo sites?
The Moz bar isn't populating information for Yahoo homepage or it's verticals (i.e. homes, autos, finance, etc.), but I can get this data for other portals like AOL or MSN. I'm specifically looking for PA, mR, and DA information, but instead I get a generic "Search Profile" bar with no page/site-specific data.
Moz Bar | | AllieBell
Is there a reason Open Site Explorer data isn't populating for this particular portal?0 -
Moz Crawl Showing Duplicate Content But It's Not?!
Unfortunately I can't give out the URL, but here's the deal... I have two URL's which have completely different content on them but are being crawled as duplicate content. Any Idea how that would happen? I'm not seeing any errors in WMT's. Has anyone seen this before? Is the duplicate content reporting based on a % of the page content matching as the same?
Moz Bar | | Swarm-SEO0 -
Moz Local | Download Template
Dear Moz I've received your email about Moz Local. A fantastic tool but it does not allow you to download a template. Clicking 'Download this template' simply reloads the page. I am testing it under incognito mode of Chrome with no add-ons Thank you!
Moz Bar | | Bio-RadAbs0 -
Moz bar issues? Does anyone else have them?
At times it's unbelievably slow and other times it doesn't show any link data. It's not just on my computer, it's on other office computers too. I'm using Google Chrome and have tried removing the mozbar and reinstalling from the chrome bar.
Moz Bar | | iresources2