How to get rid of the message "Search Engine blocked by robots.txt"
-
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest.
Regards,
Prashakth Kamath
-
Thanks Simon for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Thanks Ryan for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Hi Sagar
That was a good reply from Ryan.
Check out http://www.seomoz.org/dp/rogerbot
rogerbot is the name of the SEOmoz crawler bot, the above page has all the info you require.
Regards
Simon
-
The seomoz user agent is named rogerbot. You can read more about the SEOmoz crawl process here: http://seomoz.zendesk.com/entries/20034082-lesson-5-crawl-diagnostics
<code>User-agent: rogerbot Allow: /</code>
-
Thanks Ryan for your immediate reply.
Can you please provide name & the code of the SEOmoz Crawler that I need to enter on my file so that the SEOmoz crawls all the webpages of my website.Apart from SEOmoz Crawler I don't want any other crawler to crawl my website?Please help.Awaiting your reply.
Regards,
Prashakth Kamath
-
That error is pretty straight forward and indicates you have a robots.txt file which is blocking the crawler from accessing your site. The robots.txt file can be read by going to your site URL and adding /robots.txt to it such as www.mysite.com/robots.txt.
The file can be found in the root directory on your site's web server. Remove or alter the file to allow search engines to crawl your site. More info can be found at http://www.robotstxt.org/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt blocking Moz
Moz are reporting the robots.txt file is blocking them from crawling one of our websites. But as far as we can see this file is exactly the same as the robots.txt files on other websites that Moz is crawling without problems. We have never come up against this before, even with this site. Our stats show Rogerbot attempting to crawl our site, but it receives a 404 error. Can anyone enlighten us to the problem please? http://www.wychwoodflooring.com -Christina
Moz Pro | | ChristinaRadisic0 -
Search Volume Discepancies
I was doing research on keywords and found significant discrepancies between the search volumes on Moz and sites such as Spyfu and Semrush. What could be the reason behind these differences?
Moz Pro | | arjun97970 -
Yahoo Store Beginner with "duplicate content" errors. Can I pay for support? $$$
Hi. I have a Yahoo store that seems to have many errors. We built the site for utility knowing NOTHING about SEO. We just started with MOZ and would love to PAY someone to help get us past the beginning stages. Is there someone familiar with the Yahoo! Store format that can charge us perhaps in hourly blocks to walk us through possible solutions to issues? One issue we are having... seems to be that our subsections which contain items that are the endpoints... I know of no way to label the sections anything but an "item". I'm wondering if this might be causing the "duplicate" error because a specific item is listed both in the section and on it's own page. please help! Thom 888-567-5194
Moz Pro | | TITOJAX0 -
Historical search engine results
is there any tool that can be used to view search engine results on a given day in history for a particular term?
Moz Pro | | matmox0 -
How can I get ranking keys of competitors
Sometimes it might be clever to analyse the stategy of a competitor to see with which keys they rank successfully. For example I have a webshop, so I type in amazon.com and seomoz spits out: books, dvds, ... I can now see with what keys i wont have a chance and can think of missing keys where my chance might be. Is there a possibility to find out with seomoz?
Moz Pro | | SEOBavaria820 -
Link Blocks
Sorry, perhaps a noob question. In relation to site explorer, have also searched and unable to find any information, wondered if anyone could advise as to what "Linking C Blocks" are? Found under the "Compare Link Metrics" tab. Thanks in advance. Lee
Moz Pro | | LeeMiller0 -
How to get seomoz to re-crawl a site?
I had a lot of duplicate content issues and have fixed all the other warnings. I want to check the site again.
Moz Pro | | adamzski0 -
How do I get a list of my non-followed links
Our site has a lot of non-followed links showing up on the latest SEOMoz Competitive Link Analysis update and I would like to know how to get a list of the sites that aresettin no-follow.
Moz Pro | | oznappies0