Robots.txt blocking Moz
-
Moz are reporting the robots.txt file is blocking them from crawling one of our websites.
But as far as we can see this file is exactly the same as the robots.txt files on other websites that Moz is crawling without problems.
We have never come up against this before, even with this site.
Our stats show Rogerbot attempting to crawl our site, but it receives a 404 error.
Can anyone enlighten us to the problem please?
http://www.wychwoodflooring.com
-Christina
-
Hi Nigel
Neither, they use server side filtering.Regards- David
-
Hi David
That's great news!
As a matter of interest, where did they block it? as it's not in the Robots.txt - was in in htaccess.txt?
Regards
Nigel
-
Nigel,Thanks for the reply, the cgi-bin folder is never used by any of my sites but I put this in just as a matter of course, the folder would normally contain old cgi scripts so would not usually affect the crawling of a robot in any case.The reason for the problem turns out that our host had blocked rogerbot along with several other malicious bots, they have now lifted this block and the site is able to be crawled.- David
-
Hi Christina
I don't know how your site is set up but I can see that for some reason you are blocking access to the cgi-bin
If that directory contains files that execute php or other permissions then that may well be your problem. It's the only directory you are blocking and since I haven't seen other Robots.tx blocking it, then I would hazard a guess that this is the root of your problem.
Robots.txt
User-agent: * Disallow: /cgi-bin/ Sitemap: http://www.wychwoodflooring.com/sitemap.xml
Regards
Nigel
-
Our hosting provider has banned Rogerbot as they see it as problematic!!!!
They are a great hosting provider so this is going to be a difficult one.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz crawler not working
Hi Moz crawler keep failing on my site with the error showing : Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. I'm not sure what am I missing out.. this is my robots.txt.. i don't think Im missing anything else.. https://www.wearefutureheads.com/robots.txt can the support team help ?
Moz Pro | | teikh0 -
Hi there- Does anyone know if you can pass Adobe SC analytic data into MOZ? Rather than using GA?
Hi There, Is it possible to pass Omniture data into MOZ rather than using Google Analytics? Thanks!
Moz Pro | | GrandCircleCorp1 -
On Moz and Google analytics are showing high numbers of unknown searches.
Its getting to be 40% to 60%. Clients ask what that is and why Data Not Provided Best answers please and any advice welcome
Moz Pro | | BraveThinking0 -
Question for Moz developers - Highcharts?
So, I see that Moz is using Highcharts as it's charting display engine. What made you decide to use them instead of some of the other solutions out there, like FusionCharts or Google Charts, even creating your own home-made creation?
Moz Pro | | MrSchadow
Our company is starting over from scratch with reports/charts and are looking at other solutions than what we currently are using (fusioncharts/fusion widgets). And I wanted to get feedback on why you chose this route over any other. Thanks!!0 -
How do you get your web site recrawled with Moz without waiting for a week?
My initial crawl was screwed up because of a no follow that needed to be removed. I would like Moz to recrawl the site right away so I can find any other errors.
Moz Pro | | Ron_McCabe0 -
Moz campaign works around my robots.txt settings
My robots.txt file looks like this: User-agent: * Disallow: /*? Disallow: /search So, it should block (deindex) all dynamic URLs. If I check this url in Google: site:http://www.webdesign.org/search/page-1.html?author=47 Google tells me: A description for this result is not available because of this site's robots.txt – learn more. So far so good. Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors. One of the links is this one: http://www.webdesign.org/search/page-1.html?author=47 (the same I tested in Google and it told me that the page is blocked by robots.txt which I want) So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
Moz Pro | | VinceWicks0 -
My Moz domain authority fell last week but so did all 3 of my competitors. What could cause that?
As a loyal Moz Pro subscriber I track my site's authority, trust and links against 3 similar competitors. Last week my authority fell from 61 to 60 but all 3 competitors saw a drop in their authority that week too. Was the Moz domain authority calculation changed? Did anyone else see a drop or has something odd happened to just our market? Any ideas? Not sure if this is something I should address or just shrug and ignore. Cheers
Moz Pro | | SteveBrumpton0 -
Does the Moz Toolset offer anything like Raven's Backlink Explorer from fresh index
It seems like a useful feature where I can see what backlinks look like right now vs. the last monthly Moz dataset update. That is mostly because my site is new, but wondered if I was missing something. Thanks, Steven
Moz Pro | | sfmatthews0