Moz campaign works around my robots.txt settings
-
My robots.txt file looks like this:
User-agent: *
Disallow: /*?
Disallow: /search
So, it should block (deindex) all dynamic URLs.
If I check this url in Google:
site:http://www.webdesign.org/search/page-1.html?author=47
Google tells me:
A description for this result is not available because of this site's robots.txt – learn more.
So far so good.
Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors.
One of the links is this one:
http://www.webdesign.org/search/page-1.html?author=47
(the same I tested in Google and it told me that the page is blocked by robots.txt which I want)
So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
-
That worked, thanks!
-
Thanks Abe.
I guess I'll try this:
Useragent: Rogerbot
Disallow: /*?
Because if I use Disallow: / I'll lose my current Moz reports because Rogerbot will just ignore all my file, right?
-
Hello Vince, thank you for reaching out to us! This seems quite odd, our crawler usually obeys all robots.txt files. Let's try this. Add this code to your robots.txt:
Useragent: Rogerbot
Disallow: /
This should specifically instruct us to follow these rules. Once you have tried this, if it does not work, please send an email to help@moz.com and we will have our engineers dig in a bit further. Sorry for the inconvenience, I hope the above fix works for you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Why does Moz Crawler start with HTTP//1.1 version??
We have run Moz Pro's Site Crawl for example-domain.com Why has Moz's crawler decided to site http://www.example-domain.com (ie the non-secure version) as zero crawl depth page and report the actually live https://www.example-domain.com (ie secure HTTP2 version) as a crawl depth of 2? Surely the main live page should be the first crawled and reported as crawl depth = 0?
Moz Pro | | AKCAC0 -
Moz Domain Authority 27
Hi there, I have been reading up about Domain Authority being important to rank though I have a question about it. When trying a few google searches and using a tool to see the Moz Domain Authority i notice the following. I won't mention sites or keywords, happy to send them via private message if required. A site ranks 1st position for most keywords including the main keyword that related to their industry, a highly competitive keyword. Their Moz Domain Authority 27, Est. Visits: 265,000, Pinterest shares 72, Facebook shares 191 and unique domains that point to it are 55. Yet when I look at another site, optimising for the same keyword, Moz Domain Authority 50, Est. Visits: 60,000, Pinterest shares 35000, Facebook shares 20000 and unique domains that point to it are 521. I would have thought that the second site ranks 1st for the keyword and the first one mentioned which is in 1st position I would have expected to be on page two or three. as I scroll down the first page for the term, all sites except for one have higher DA, visits, shares and so on. If a high PA is required can someone help me to understand the above example? Whn a low PA site ranks much better for a primary keyword than all other sites that have a higher PA than it?
Moz Pro | | IvanaDaulay1 -
Meta Robots query
Hi guys, I was ranking really well on my home page for certain keywords which has all dropped pretty dramatically over the last 3/4 weeks - I think the issue is since since the configuration of Yoast SEO Wordpress plugin. In March (when my rankings were strong) my crawl test showed the top data in the attached image, and in May (now the rankings have dropped severly) they show the bottom data. I don't fully understand canonical and Meta Robots so I am hoping someone can shed some light on the following points. 1. Will the change result in my loss of rankings.
Moz Pro | | RocketStats
2. How can I put it back to how it was in March? PS. I haven't had any Google penalties. Thanks,
Joshua RfTar0 -
[Moz Help] Re: Trying to add a valid URL into MOZ account
See below and pls let us know what we have to do solve this : | | Joel Day (Moz Help) Mar 07 05:03 PM Hey Tracy, It looks like there's a redirect loop on your site. greatwesternflooring.com redirects to www.greatwesternflooring.com/ which in turn 302 redirects back into itself. You'll likely need to fix the redirect before you can continue configuring the campaign. 🙂 Thanks!
Moz Pro | | Britewave
Joel. Moz
t: @HelpWizard | | | Tracy Mar 07 03:14 PM I sent an email, and this is the response I got. The help forum sent me here, so here I am 🙂 An answer was posted to this question:
Question I have a valid URL greatwesternflooring.com, but when I try to add this campaign I get an "opps" message telling me it's not a valid URL. Can you help me? Answer
This looks like a bug. Please reach out to us via support so that we can forward this along to our Developers for review. Thanks!(https://moz.com/help/contact)
See where this question was originally asked. |0 -
Why isn't me SellingRestaurants campaign operating???
I resign a week ago and I'm still waiting for my SellingRestaurants.com campaign to run???
Moz Pro | | restaurantbroker0 -
Moz WordPress Plugin?
WordPress is currently 18% of the Internet. Given its huge footprint, wouldn't it make sense for Moz to develop a WP plugin that can not only report site metrics, but help fix and optimize site structure directly from within the site? Just curious - I can't be the only one who wonders if I'm implementing Moz findings/recommendations correctly given the myriad of WP SEO plugins, authors, implementations.
Moz Pro | | twelvetwo.net4 -
Does SEOmoz work for fully secure sites (https)?
We have created a large file sending service (https://digitalpigeon.com) and for obvious reasons we have secured the entire site (we used to have a trial file send form on the homepage so it made sense to secure everything with 301 redirects from http to https) Because we, somewhat naively, started out with a memorable name and domain - Digital Pigeon - where all of our competitors just keyword stuffed their domains we don't really rank for any useful keywords yet. For me, building domain authority via links is more important right now that any on page SEO that is going to be trumped by google's infatuation with domain keywords anyway. Here's the problem though, it seems SEOmoz and even Google Webmaster Tools and Bing Webmaster Tools all fail to take into account inbound links for both http and https for any reports or analysis! I'm wondering if utilising SEOmoz PRO for keeping tabs on our MozRank is pointless if it's not accounting for all inbound links
Moz Pro | | DigitalPigeon0 -
How do I get back my archived campaigns??
Hi there, Can anyone help! I archived a campaign a month ago and now I want it back - can anyone help? How do I get back my archived campaign? Or do I need to start again?? Thanks Gareth
Moz Pro | | GAZ090
