Moz "Crawl Diagnostics" doesn't respect robots.txt
-
Hello, I've just had a new website crawled by the Moz bot. It's come back with thousands of errors saying things like:
- Duplicate content
- Overly dynamic URLs
- Duplicate Page Titles
The duplicate content & URLs it's found are all blocked in the robots.txt so why am I seeing these errors?
Here's an example of some of the robots.txt that blocks things like dynamic URLs and directories (which Moz bot ignored):Disallow: /?mode=
Disallow: /?limit=
Disallow: /?dir=
Disallow: /?p=*&
Disallow: /?SID=
Disallow: /reviews/
Disallow: /home/Many thanks for any info on this issue.
-
Hi Si, has this issue been resolved?
-
Hey Si,
Thanks for writing in. It doesn't seem that we are having an overarching issue with our crawler ignoring robots.txt files so I did some research in Google Webmaster Tools and it looks like most crawlers require an asterisk in the disallow directive to recognize that all pages of a dynamic URL are being disallowed. If you look in the "Pattern Matching" section of this resource here: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449, that should give you more information about setting up the robots.txt with the correct disallow directives to block those pages.
If you add in the astrisk to the disallow directive and you are still seeing these pages crawled, it would help if you sent in an email with your campaign information to our support desk at help@moz.com so we can have our engineers look into this more directly.
I hope this helps.
Chiaryn
-
If you have an "index,(no)follow" meta on those pages I think they will be crawled even though you have them blocked in robots.txt. So by adding "noindex" on those pages it might work as you want it to.
-
Is the / actually in the URL at that spot? Or is your link like http://www.example.com/abcd?p=147
If you give an example full URL that includes one of your blocked dynamic URLs we can take a better look. If your robots is setup correctly, it shouldn't find that stuff but give us more info if you're able.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I 'sign in' to the Moz Bar?
It's installed, I can see links etc as highlighted - but it won' t let me "sign in". This 20 second video explains: https://www.screencast.com/t/3kEjQFkTHZv Suggestions? Or shall I just ignore? Paul Barrs
Moz Bar | | PaulBarrs0 -
Is Moz more accurate than Ahrefs ?
When I check my this website https://joinpakarmy.com.pk/ on Moz it gives me DA 3
Moz Bar | | joinpakarmy234
But when I check it out on Ahrefs, it tells me there is no authority for this site.
So who is more accurate in the result?
thanks0 -
MozBar Issues? Can't get info even logged in
Anyone having issues with the Moz bar? Lately no matter how many times I log in, getting data is difficult as it keeps asking me to create an acct. or log in. Even logged into Moz and on Q&A it is asking me to log in. THanks
Moz Bar | | RobertFisher0 -
MOZ Staff: Timeline for supporting SNI?
We have moved our blog to Amazon Web Services, and our website is soon to follow. For better or worse, AWS uses SNI, which MOZ doesn't currently support. Here are some recent forum posts about it: https://moz.com/community/q/804-server-error-crawling-https https://moz.com/community/q/804-https-ssl-error This makes MOZ much, much less useful to me. MOZ staff, you have a timeline for when you'll implement support for SNI?
Moz Bar | | Atomic-Object2 -
Moz Crawl Test says pages have no internal links
Greetings, I am working on a website, https://www.nasscoinc.com, and ran a Moz Crawl Test on it. According to the crawl test, only 2 of the website's hundreds of pages are receiving internal links. When I run a similar test on the site using Screaming Frog, I see that most of the pages have at least one internal link. I'm wondering if anyone has seen this before with the crawl test; and there is a way to get the crawl test to see the internal links? Thanks!
Moz Bar | | TopFloor0 -
Can't download mozbar
Fine, I'm an idiot. But when i get that Chrome apps page with mozbar on it, there's nothing clickable (that I can see) that actually downloads the thing. I've gotten it before, but now I can't find it. Is it hiding. Why doesn't it just sit in my dock? Any ideas. -W
Moz Bar | | wrconard0 -
How can I find the old ERRORS and WARNINGS report in the NEW Moz design?
I'm looking for a complete list of errors and bugs that need to be fixed within a website. I used to use the MAIN tool (at least it seemed it was the most popular) but now that its just MOZ.com I can't seem to find that great report. It had data such as: 1. List of pages with Title Tags too long 2. List of pages with Description Tags too long 3. List of RED errors and YELLOW warnings, BLUE somethings... etc... Ring a bell? I LOVED this report, where can I find this data? Thanks! Derek
Moz Bar | | DerekM42420 -
Whats wrong with the typography of moz?
Seems moz is testing webfonts? On Chrome its hard to read not to say horrible. On FF its a little better.Or is this just on my computer?
Moz Bar | | inlinear0