How to get rid of the message "Search Engine blocked by robots.txt"
-
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest.
Regards,
Prashakth Kamath
-
Thanks Simon for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Thanks Ryan for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Hi Sagar
That was a good reply from Ryan.
Check out http://www.seomoz.org/dp/rogerbot
rogerbot is the name of the SEOmoz crawler bot, the above page has all the info you require.
Regards
Simon
-
The seomoz user agent is named rogerbot. You can read more about the SEOmoz crawl process here: http://seomoz.zendesk.com/entries/20034082-lesson-5-crawl-diagnostics
<code>User-agent: rogerbot Allow: /</code>
-
Thanks Ryan for your immediate reply.
Can you please provide name & the code of the SEOmoz Crawler that I need to enter on my file so that the SEOmoz crawls all the webpages of my website.Apart from SEOmoz Crawler I don't want any other crawler to crawl my website?Please help.Awaiting your reply.
Regards,
Prashakth Kamath
-
That error is pretty straight forward and indicates you have a robots.txt file which is blocking the crawler from accessing your site. The robots.txt file can be read by going to your site URL and adding /robots.txt to it such as www.mysite.com/robots.txt.
The file can be found in the root directory on your site's web server. Remove or alter the file to allow search engines to crawl your site. More info can be found at http://www.robotstxt.org/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
PA Score of 30+ to get Google Features
I was told that my pages would not be awarded a Google Feature until the Page Authority reaches 30. I had Features previously, but no longer. The PA for the URL is currently 22 https://heritageprintingcharlotte.com/products/meter-boards.php I have edited the content, built links and added schema markup to this particular page (and others) I'm looking for thoughts on the 30+ and how to improve PA. TY KJr
Moz Pro | | KevnJr0 -
What is error message:social_account.no_method?
When trying to add a twitter account to track, I received the following error message:social_account.no_method - has anyone else received this message?
Moz Pro | | solutionbuilt.com0 -
What does the "Internal Links" data in the Keyword Difficulty SERP report represent?
What does the "Internal Links" data in the Keyword Difficulty SERP report represent? Thank you! QocBS
Moz Pro | | richpalpine0 -
How can i get seomoz to crawl a campaign on demand
hi how can i get seomoz to crawl a campaign on demand instead of on a weekly basis? For example i have corrected some error warnings and on page elements and would like it to re crawl the site sooner to see how the corrections have worked? thanks
Moz Pro | | Bristolweb0 -
Will canonical tag get rid of duplicate page title errors?
I have a directory on my website, paginated in groups of 10. On page 2 of the results, the title tag is the same as the first page, as it is on the 3rd page and so on. This is giving me duplicate page title errors. If i use rel=canonical tags on the subsequent pages and href the first page of my results, will my duplicate page title warnings go away? thanks.
Moz Pro | | fourthdimensioninc0 -
Localized Search Results
I provide SEO for local service providers therefore I am mostly interesting in ranking highly in the "localized results" SERPs. Is anyone familiar with any good tools, infographics, blogs, or webinars that could be of benefit? How big of a radius do the localized results usually encompass? Are the keyword ranking reports from SEOmoz even relevant for reporting purposes? Should I include the city name (i.e. "Seattle Doctor", "Doctor in Seattle", "Doctor Seattle", etc.) in my keyword research or is this becoming less relevant? Any insight is welcome and appreciated! Thanks in advance!!
Moz Pro | | GoChiroPractice0 -
Link Blocks
Sorry, perhaps a noob question. In relation to site explorer, have also searched and unable to find any information, wondered if anyone could advise as to what "Linking C Blocks" are? Found under the "Compare Link Metrics" tab. Thanks in advance. Lee
Moz Pro | | LeeMiller0 -
Why is Roger crawling pages that are disallowed in my robots.txt file?
I have specified the following in my robots.txt file: Disallow: /catalog/product_compare/ Yet Roger is crawling these pages = 1,357 errors. Is this a bug or am I missing something in my robots.txt file? Here's one of the URLs that Roger pulled: <colgroup><col width="312"></colgroup>
Moz Pro | | MeltButterySpread
| example.com/catalog/product_compare/add/product/19241/uenc/aHR0cDovL2ZyZXNocHJvZHVjZWNsb3RoZXMuY29tL3RvcHMvYWxsLXRvcHM_cD02/ Please let me know if my problem is in robots.txt or if Roger spaced this one. Thanks! |0