How to get rid of the message "Search Engine blocked by robots.txt"
-
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest.
Regards,
Prashakth Kamath
-
Thanks Simon for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Thanks Ryan for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Hi Sagar
That was a good reply from Ryan.
Check out http://www.seomoz.org/dp/rogerbot
rogerbot is the name of the SEOmoz crawler bot, the above page has all the info you require.
Regards
Simon
-
The seomoz user agent is named rogerbot. You can read more about the SEOmoz crawl process here: http://seomoz.zendesk.com/entries/20034082-lesson-5-crawl-diagnostics
<code>User-agent: rogerbot Allow: /</code>
-
Thanks Ryan for your immediate reply.
Can you please provide name & the code of the SEOmoz Crawler that I need to enter on my file so that the SEOmoz crawls all the webpages of my website.Apart from SEOmoz Crawler I don't want any other crawler to crawl my website?Please help.Awaiting your reply.
Regards,
Prashakth Kamath
-
That error is pretty straight forward and indicates you have a robots.txt file which is blocking the crawler from accessing your site. The robots.txt file can be read by going to your site URL and adding /robots.txt to it such as www.mysite.com/robots.txt.
The file can be found in the root directory on your site's web server. Remove or alter the file to allow search engines to crawl your site. More info can be found at http://www.robotstxt.org/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best Way To Write H1 to Get Max CTR?
Can anyone please share some guidelines regarding how to write the H1 subject of the post which can generate the maximum CTR.?. I noticed even when i able to rank on the featured snippet with good enough search volume the CTR is too low. Anyone share some expert advice please?
Moz Pro | | ndhsne450 -
Is BigCommerce a good CMS for Improving Search Visibility for our E-Commerce Business?
We have recently moved from Wordpress to BigCommerce and have noticed a decline in our Search Visibility score on MOZ. Following the recommendations of IT consultants, we were under the impression that moving to BigCommerce would significantly improve our Search Visibility. We were wondering if there is anyone out there who is currently using or have used BigCommerce and had similar issues? Any personal reviews/recommendations would be much appreciated! 🙂 Thanks!
Moz Pro | | lisakay0 -
Is a new website the only way to get better rankings?
Our website is https://www.franklin-bell.com.au. For a while now we've been trying to get a few of our pages on there to get a better rank on google. A local SEO agency told us that some of our pages are not like "the modern standard" for web pages, but our page score for the keywords we want is 90% and higher. Do we really need to completely rebuild the website (it was built in 2014) in order to get better rankings? Or is it sufficient to have the 97% page score and then just get some back links and social media referrals etc. Thanks in advance.
Moz Pro | | franklin-bell0 -
Should I set blog category/tag pages as "noindex"? If so, how do I prevent "meta noindex" Moz crawl errors for those pages?
From what I can tell, SEO experts recommend setting blog category and tag pages (ie. "http://site.com/blog/tag/some-product") as "noindex, follow" in order to keep the page quality of indexable pages high. However, I just received a slew of critical crawl warnings from Moz for having these pages set to "noindex." Should the pages be indexed? If not, why am I receiving critical crawl warnings from Moz and how do I prevent this?
Moz Pro | | NichGunn0 -
How do I get back my archived campaigns??
Hi there, Can anyone help! I archived a campaign a month ago and now I want it back - can anyone help? How do I get back my archived campaign? Or do I need to start again?? Thanks Gareth
Moz Pro | | GAZ090 -
Why does SEOMoz crawler ignore robots.txt?
The SEOMoz crawler ignores robots.txt It also "indexes" pages marked as noindex. That means it is filling up the reports with things that don't matter. Is there any way to stop it doing that?
Moz Pro | | loopyal0 -
Duplicate content due to "Email a Friend" and "PhotoGallery"
Crawl Diagnostics gives me 1650 duplicate page content errors. 800+ are for photo gallery, the link upon which my large image is shown. 800+ are for the Email a Friend form page. Presumably SEOMoz's tool is getting there by following the link inside a product page, which as you can see will differ only by the ProductCode of every product.
Moz Pro | | jotham2
www.completemobilehomesupply.com/PhotoGallery.asp?ProductCode=anchor101
www.completemobilehomesupply.com/EmailaFriend.asp?ProductCode=Shutter001 1 - Is it critical to fix?
I am assuming Yes, but if for some reason the answer is No, please share. 2 - Any idea on how to fix?
The site uses Volusion, fyi, so it may be a limiting factor of the platform. Thank you for your time.
Also if this topic has been previously covered, please link and I'll read there instead.0 -
How to get seomoz to re-crawl a site?
I had a lot of duplicate content issues and have fixed all the other warnings. I want to check the site again.
Moz Pro | | adamzski0