Moz campaign works around my robots.txt settings
-
My robots.txt file looks like this:
User-agent: *
Disallow: /*?
Disallow: /search
So, it should block (deindex) all dynamic URLs.
If I check this url in Google:
site:http://www.webdesign.org/search/page-1.html?author=47
Google tells me:
A description for this result is not available because of this site's robots.txt – learn more.
So far so good.
Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors.
One of the links is this one:
http://www.webdesign.org/search/page-1.html?author=47
(the same I tested in Google and it told me that the page is blocked by robots.txt which I want)
So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
-
That worked, thanks!
-
Thanks Abe.
I guess I'll try this:
Useragent: Rogerbot
Disallow: /*?
Because if I use Disallow: / I'll lose my current Moz reports because Rogerbot will just ignore all my file, right?
-
Hello Vince, thank you for reaching out to us! This seems quite odd, our crawler usually obeys all robots.txt files. Let's try this. Add this code to your robots.txt:
Useragent: Rogerbot
Disallow: /
This should specifically instruct us to follow these rules. Once you have tried this, if it does not work, please send an email to help@moz.com and we will have our engineers dig in a bit further. Sorry for the inconvenience, I hope the above fix works for you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why my website internal links on moz is 0?
My Web site: https://www.basictrailers.com.au/ I have been using Moz for almost 3 months, from the start to now, the internal links are always 0. However, if I check the links on my Google Search Console, it is over 1500. So what is the problem? If Moz is trying to make its algorithm similar to Google, why that they even cannot find my internal links. That is ridiculous. 0fhTnQG
Moz Pro | | Lilycharlie2 -
Why does Moz show 302s that I previously resolved?
So Moz was flagging thousands of 302 errors in the Redirect Issues section. It was due to the URL extensions containing /directory/currency/switch/currency/ I added this to my Robots.txt file as I don't want these indexed. I marked the 302 errors as fixed and after the next crawl they came back and I now get the message: One or more previously fixed issues continue to persist. We have found one or more issues that were marked as fixed previously in the last crawl. Do I just ignore the errors or is there something wrong that I may be doing? Any help would be much appreciated. Thank you.
Moz Pro | | lbagley0 -
Should I set blog category/tag pages as "noindex"? If so, how do I prevent "meta noindex" Moz crawl errors for those pages?
From what I can tell, SEO experts recommend setting blog category and tag pages (ie. "http://site.com/blog/tag/some-product") as "noindex, follow" in order to keep the page quality of indexable pages high. However, I just received a slew of critical crawl warnings from Moz for having these pages set to "noindex." Should the pages be indexed? If not, why am I receiving critical crawl warnings from Moz and how do I prevent this?
Moz Pro | | NichGunn0 -
Question for Moz developers - Highcharts?
So, I see that Moz is using Highcharts as it's charting display engine. What made you decide to use them instead of some of the other solutions out there, like FusionCharts or Google Charts, even creating your own home-made creation?
Moz Pro | | MrSchadow
Our company is starting over from scratch with reports/charts and are looking at other solutions than what we currently are using (fusioncharts/fusion widgets). And I wanted to get feedback on why you chose this route over any other. Thanks!!0 -
What keywords should I use in my campaign.
I am a chiropractor. I only compete against web sites in my town. Should my key words be neck pain, back pain? Or should it be neck pain Bloomington Illinois and low back pain Bloomington Illinois? I am referring to the campaigns I just started. (I am new)
Moz Pro | | Bob550 -
Campaign report errors
one of the heavily noted errors for the first crawl of our domain was duplicate titles. I did not see a list of the pages and their current titles, but i am pretty sure it is somewhere to be found. That would help focus the work to do. Am I wrong about that?
Moz Pro | | Jacog0 -
What Does the SEO Moz Crawler Take Into account?
I'm working on a page that has links from some decent pages pointing to it, but a lot of them are low-value blog comments. So I'm pretty sure that its Page Authority is higher than it should be, compared to where it's ranking. Does SEO moz take the type of link into account? i.e. if it's a footer link, blog comment, or forum signature link; this should carry less weight than a link in the content of the page itself, as it does with Google.
Moz Pro | | seanmccauley0 -
Limit of 3 competitors per campaign
Hello, Can I just check why the number of competitors per campaign is limited to 3? ie. We have up to 20 URLs we want to monitor for the same keywords as us. I understand I can add another campaign to track another 4x URLs, but this will not show those in relation to our main URL/campaign. We would prefer to have space for 12 competitors rather than 12 campaigns. If we upgrade to Elite is the number increased? Might be missing something here, as we are new to SEOMoz, so grateful for any advice. SEOMoz is really great, we have learned so much already. Rich
Moz Pro | | STL0