Moz campaign works around my robots.txt settings
-
My robots.txt file looks like this:
User-agent: *
Disallow: /*?
Disallow: /search
So, it should block (deindex) all dynamic URLs.
If I check this url in Google:
site:http://www.webdesign.org/search/page-1.html?author=47
Google tells me:
A description for this result is not available because of this site's robots.txt – learn more.
So far so good.
Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors.
One of the links is this one:
http://www.webdesign.org/search/page-1.html?author=47
(the same I tested in Google and it told me that the page is blocked by robots.txt which I want)
So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
-
That worked, thanks!
-
Thanks Abe.
I guess I'll try this:
Useragent: Rogerbot
Disallow: /*?
Because if I use Disallow: / I'll lose my current Moz reports because Rogerbot will just ignore all my file, right?
-
Hello Vince, thank you for reaching out to us! This seems quite odd, our crawler usually obeys all robots.txt files. Let's try this. Add this code to your robots.txt:
Useragent: Rogerbot
Disallow: /
This should specifically instruct us to follow these rules. Once you have tried this, if it does not work, please send an email to help@moz.com and we will have our engineers dig in a bit further. Sorry for the inconvenience, I hope the above fix works for you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Make Moz reindex the pages for keywords
Hi Moz says we're 18th for a phrase but that's for the page /himonsia-generators because that was the highest ranking page. If you search now for "heavy range generators" we're jumping between 10-12th position but it's for the page /products/generators - which is correct. How can I make Moz "Reindex" the pages?
Moz Pro | | RichardAskew0 -
How do I set-up a 301 redirect?
Moz alerted me to two 404 errors on our website. I am now trying to place a 301 redirect on them. How would I go about accomplishing?
Moz Pro | | EverlastingChanges0 -
Spam score by moz
como puedo estar seguro de que los enlaces toxicos que informa la herramienta moz spam score, son realmente malos para mi seo? Algunos de ellos parecen totalmente buenos! gracias!! as I can be sure that the toxic links moz tool reports the spam score, they are actually bad for my SEO? Some of them seem totally good!
Moz Pro | | Sbm20080 -
Error in Moz duplicate content reports
Hi - I've run the Moz campaign on a client's site. Moz is saying that there are duplicate content errors, and when I look at the errors it is showing that they are all to do with the non-www URLs having being duplicated in the www form of the URLs. However this is not the case - all the non-www URLs are all 301 redirected to the www URLs. Is this an error in the Moz tool? Has anybody experienced something similar?
Moz Pro | | rorynatkiel0 -
Campaign set-up: root domain vs. subdomain
Hi, I have a question about the proper way to set up a new SEOmoz campaign for a simple 5-page website (home, about, services, blog, contact). If I'm tracking only www.domain.com, does it make a difference if I set it up to track just that subdomain or should I track it at the root domain? Is it better to use the root domain? Are there pros or cons to using one over the other? If the website is expanded in the future, any future pages will most likely be on the same subdomain (e.g., www.domain.com/page). Thanks in advance! Carolina
Moz Pro | | csmm0 -
Drop in number of Pages crawled by Moz crawler
What would cause a sudden drop in the number of pages crawled/accessed by the Moz crawler? The site has about 600 pages of content. We have multiple campaigns set up in our Pro account to track different keyword campaigns- but all for the same domain. Some show 600+ pages accessed, while others only access 7 pages for the same domain. What could be causing these issues?
Moz Pro | | AllaO0 -
Campaign warnings and recommendations
How crucial is it to follow / fix the campaign warnings. For example, even though our website is ranking well the campaign report is stating that we've too many links on the majority of pages and the title element is too long. My question is if I make changes based on these recommendations, do you think that we will see a dramatic improvement in our rankings ?
Moz Pro | | NeilTompkins0