Moz campaign works around my robots.txt settings
-
My robots.txt file looks like this:
User-agent: *
Disallow: /*?
Disallow: /search
So, it should block (deindex) all dynamic URLs.
If I check this url in Google:
site:http://www.webdesign.org/search/page-1.html?author=47
Google tells me:
A description for this result is not available because of this site's robots.txt – learn more.
So far so good.
Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors.
One of the links is this one:
http://www.webdesign.org/search/page-1.html?author=47
(the same I tested in Google and it told me that the page is blocked by robots.txt which I want)
So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
-
That worked, thanks!
-
Thanks Abe.
I guess I'll try this:
Useragent: Rogerbot
Disallow: /*?
Because if I use Disallow: / I'll lose my current Moz reports because Rogerbot will just ignore all my file, right?
-
Hello Vince, thank you for reaching out to us! This seems quite odd, our crawler usually obeys all robots.txt files. Let's try this. Add this code to your robots.txt:
Useragent: Rogerbot
Disallow: /
This should specifically instruct us to follow these rules. Once you have tried this, if it does not work, please send an email to help@moz.com and we will have our engineers dig in a bit further. Sorry for the inconvenience, I hope the above fix works for you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does increase in DA increase Moz Trust score?
I have checked many website which in my office. Have a confusion Does increase in DA increase Moz Trust score? I have a website https://emocrest.com/ which is new one and Moz Trust score as of now is 1 only. Well, most of the projects I'm handling in our office having 2,3,4 in the range. How this MT value is calculated and does it got any relation with the change in Domain Authoiry or the page authority of a website?
Moz Pro | | nazfazy1 -
Moz Website Grader Export Omits Check Graphics
So, I love the Grader, but when I export to PDF the pretty little graphics go away. Is this correctable?
Moz Pro | | ClickMonsterIM0 -
Keyword Stuffing - MOZ On-Page Grader
We sell a great number of insulation products, many of which are produced by individual manufacturers. On the page identified below the Keyword "Kingspan" is repeated numerous times as these items are included in our online shop. However, the many mentions of Kingspan are recorded in the HTML5 Source Code, rather than an external database. When I used the MOZ On-Page Grader, using the keywords "Kingspan" I was surprised to achieve an "A" Grade! I know I shouldn't be complaining, but I am wondering why the significant repetition of the word "Kingspan" has not negatively impacted my score? http://www.just-insulation.com/001-eshop/buy-kingspan-thermapitch-thermawall-thermafloor-insulation-boards.html
Moz Pro | | JustInsulation0 -
Do I need a new moz campaign for a subdomain?
Will moz automatically track my new subdomain or do I need to set up a new campaign for it?
Moz Pro | | SamCUK0 -
Multiple Sites/Internal Pages Campaign
Can MOZ do reports/ranks/campaigns separately for each of our sites, then do separate keyword campaigns for specific internal content pages of each of the websites? For a law firm with both a defense and family law sites. We have multiple pages within each site, we will need to do separate and individual campaigns for assault, burglary, traffic tickets, ect. in our defense site. And we'll need other specific campaigns for other specific content in out family law site. Will Moz be able to accommodate our needs with their $99/mo plan or is that service only available for higher packages? (or is it even possible in ANY of their packages???) Thanks.
Moz Pro | | Wallin_Klarich0 -
SEO moz poor rankings reporting
How are we supposed to send ranking reports to clients if the rankings are constantly reporting false metrics This is not the first time this has happened, but the rankings are WAAAAAAY off. What is the point of reporting rankings that are not accurate? What is the possible cause of this. Is there a solution? What is SEOmoz doing to improve their ranking programs. Our client actually ranked #2 for "northern Virginia HVAC contractors" when moz had us not in the top 50! How are we supposed to bank on information that is so consistently unreliable? northernvirginiahvaccontractors.png northernvirginiahvaccontractorsrank.png
Moz Pro | | imageworks-2612900 -
Campaign report errors
one of the heavily noted errors for the first crawl of our domain was duplicate titles. I did not see a list of the pages and their current titles, but i am pretty sure it is somewhere to be found. That would help focus the work to do. Am I wrong about that?
Moz Pro | | Jacog0 -
Keyword Difficulty Tool not working!
Hello everyone, I quite new to the pro service and I am still on my free trial and I have to say that it is perfect! However, I haven't been able to use the keyword difficulty tool for a couple of days as it keeps saying that there is a problem and that I should try again in 20 minutes. I tried several times and 9 out of 10 times I get this message. I have only managed to get a couple of keywords analyzed. Keep in mind that most of the times I am not using English terms but since it worked in the past I would think that this is not the problem. Any thoughts on how to fix this issue? Thanks in advance!
Moz Pro | | Somondo1