Robots.txt blocking Moz
-
Moz are reporting the robots.txt file is blocking them from crawling one of our websites.
But as far as we can see this file is exactly the same as the robots.txt files on other websites that Moz is crawling without problems.
We have never come up against this before, even with this site.
Our stats show Rogerbot attempting to crawl our site, but it receives a 404 error.
Can anyone enlighten us to the problem please?
http://www.wychwoodflooring.com
-Christina
-
Hi Nigel
Neither, they use server side filtering.Regards- David
-
Hi David
That's great news!
As a matter of interest, where did they block it? as it's not in the Robots.txt - was in in htaccess.txt?
Regards
Nigel
-
Nigel,Thanks for the reply, the cgi-bin folder is never used by any of my sites but I put this in just as a matter of course, the folder would normally contain old cgi scripts so would not usually affect the crawling of a robot in any case.The reason for the problem turns out that our host had blocked rogerbot along with several other malicious bots, they have now lifted this block and the site is able to be crawled.- David
-
Hi Christina
I don't know how your site is set up but I can see that for some reason you are blocking access to the cgi-bin
If that directory contains files that execute php or other permissions then that may well be your problem. It's the only directory you are blocking and since I haven't seen other Robots.tx blocking it, then I would hazard a guess that this is the root of your problem.
Robots.txt
User-agent: * Disallow: /cgi-bin/ Sitemap: http://www.wychwoodflooring.com/sitemap.xml
Regards
Nigel
-
Our hosting provider has banned Rogerbot as they see it as problematic!!!!
They are a great hosting provider so this is going to be a difficult one.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Why is Moz reporting 308s as 302s?
Got a looooong list of redirect issues in my crawl for a new client, all reported as 302s but as far as I can see they are all 308s... which is perfectly fine, right, or have I missed a memo? They even confirm the 308 status in the moz detail.
Moz Pro | | Algorhythm_jT0 -
How to utilize Moz during website relaunch?
Hi folks - we're relaunching our website and have been sure to make it SEO optimized -- descriptive copy, new readable URLs, keyword-laden page metadata, and the like. However, I'm new to Moz Pro and unsure how to utilize it during this transition. My questions for those of you -- assuming many -- who have gone through a redesign: **How did you use Moz to track the success of your redesigned website? What do you wish you'd done? ** Any campaigns or resources you can suggest will be helpful, as I am new to the functionality!
Moz Pro | | meghanennes0 -
Moz is treating my pages as duplicate content but the pages have different content in reality
Attached here is a screenshot of links with duplicate content. Here are some links that is also based on the screenshot http://federalland.ph/construction_updates/paseo-de-roces-as-of-october-2015 http://federalland.ph/construction_updates/sixsenses-residences-tower-2-as-of-october-2015/ http://federalland.ph/construction_updates/sixsenses-residences-tower-3-as-of-october-2015 The links that I have placed here have different content. So I don't why they are treated as duplicates BWWJuvQ
Moz Pro | | clestcruz0 -
Moz doesn’t show the right facebook likes.
Hi moz help, I found out that in opensiteexplorer.org our Facebook likes counts zero, while my facebook account actually counts more than 600 likes. How is this possible? Did I not couple my facebook account correctly to my website? The address is http://www.onlinefietsonderdelenkopen.nl and the Facebook page is https://www.facebook.com/onlinefietsonderdelenkopen
Moz Pro | | Seeders0 -
C-Block domains OSE
hi all quick question regarding c-block domains OSE tells me we have 70 c-block domains with a total 130 root domains, is it telling us 70 root domains re c-blocks as this is near impossible for us are c blocks listed as root domains or just links
Moz Pro | | Will_Craig0 -
Moz Crawl Test: WordPress sites with and without /feed and /trackback entires?
I have multiple WP websites and on some of the websites, on my Moz Crawl test, I see an entry for every blog post but also entries for /feed and /trackback for that single blog post. For example, www...com/someArticle www....com/someArticle/feed www...com/someArticle/trackback 1. Can anyone explain why the Crawl test is picking up the /feed and /trackback items? Is it simply because they are 301 redirects to the original post (www...com/someArticle)? 2. What setting(s) in WordPress are making this information appear? Or is it just that the site(s) that have the /feed and /trackback are displaying "normal" behavior for a WP site with a lot of trackbacks and feed entires? 3. Should /fee and /trackback, as well as /author be blocked in robots.txt? Thanks in advance for your advice and input!
Moz Pro | | Titan5520 -
Simply Report for new requests with Moz Pro Plus
Hi all I'd like to know how can I construct a brief report to send to my potential customers when they first write me asking for general informations and evaluation about their websites and certain keywords for which they want to have a great rank. I'd like to reply them with a short email, with a basic report attached, in which showing them some little details about their (and eventually their competitors) ranking and/or their website situation. It's just a brief professional introduction to our seo and web marketing services. Now I'm doing it manually or with other seo tools, but I have just bought a Moz Pro Plus annually version and I'd like to use Moz for this activity. For example, when I first insert a new campaign, I don't see ranking for days so I can't send rankinh report. Do you have any advice? Thanks.
Moz Pro | | Penny10 -
What do the dates refer to in seo moz reports
question is in the title - new trainee asked me and couldn't actually answer!
Moz Pro | | Highlandgael0