Moz campaign works around my robots.txt settings
-
My robots.txt file looks like this:
User-agent: *
Disallow: /*?
Disallow: /search
So, it should block (deindex) all dynamic URLs.
If I check this url in Google:
site:http://www.webdesign.org/search/page-1.html?author=47
Google tells me:
A description for this result is not available because of this site's robots.txt – learn more.
So far so good.
Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors.
One of the links is this one:
http://www.webdesign.org/search/page-1.html?author=47
(the same I tested in Google and it told me that the page is blocked by robots.txt which I want)
So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
-
That worked, thanks!
-
Thanks Abe.
I guess I'll try this:
Useragent: Rogerbot
Disallow: /*?
Because if I use Disallow: / I'll lose my current Moz reports because Rogerbot will just ignore all my file, right?
-
Hello Vince, thank you for reaching out to us! This seems quite odd, our crawler usually obeys all robots.txt files. Let's try this. Add this code to your robots.txt:
Useragent: Rogerbot
Disallow: /
This should specifically instruct us to follow these rules. Once you have tried this, if it does not work, please send an email to help@moz.com and we will have our engineers dig in a bit further. Sorry for the inconvenience, I hope the above fix works for you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Fluctuation in MOZ DA
HI, i want to know how MOZ DA increase or decrease. My Website: https://yesornogenerator.com/ has 8 DA according to MOZ but sometimes it goes back to 5 and then increase to 8. There is no change on backlinks so how this happens.
Moz Pro | | TOFHA0545150 -
Looking for a MOZ pro to hire
Just got MOZ pro and need someone to look things over and give me a list of things to do first. When everyone claims to be an expert online. Any suggestions on who to hire or use?
Moz Pro | | Audio-Bible1 -
Bing SEM Traffic showing up as Organic in Moz?
Hi there, Everyone! First time poster. 🙂 One of my clients has an ads string showing up in organic search? It is: s=bing_search&p=none . If I block it in the robots text, then I won't see any traffic results.
Moz Pro | | TopGrowthHacker.com0 -
Campaign Setup-GA account
When setting up a campaign- in select a Google account it does not show the subdoman account. The drop down has other accounts we have but not the one we want.
Moz Pro | | RNK0 -
More complete campaign reports
SEOMoz campaigns include a lot more data than can be sent via email automatically with the custom reporting feature. So what do people do to send that data to clients? Do you not include it? Or export reports manually and send them on? Or something else?
Moz Pro | | antdesign0 -
Can I do a campaign for just a page?
We've been doing a lot of building and work on just one category page, but when i try to put it in the campaign it won't let me do any url that has a sub folder like www.mainsite.com/keyword-page. I can only do www.mainsite.com, and when i select the other campaign options like root domain or sub folder, roger pops up with an error. Is anyone else having this problem?
Moz Pro | | anchorwave0 -
Campaign Manager Keyword Rankings all messed up???
My campaign in SEO Moz Campaign Manager has just ran it's weekly crawl, and all my keyword rankings are messed up. 18 keywords have apparently dropped by massive amounts, but when I do a physical search, they haven't moved. Also Keyword Tracker agrees with the physical search, so why is the Campaign Manager showing the wrong rankings all of a sudden??
Moz Pro | | gdavies090319770 -
301 redirect in SEOMoz campaigns tool
I did a 301 redirect to another domain and I would like to change the domain name in SeoMoz campaigns tool to continue to track the keywords, is it possible ?
Moz Pro | | mhenriques0