How to get rid of the message "Search Engine blocked by robots.txt"
-
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest.
Regards,
Prashakth Kamath
-
Thanks Simon for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Thanks Ryan for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Hi Sagar
That was a good reply from Ryan.
Check out http://www.seomoz.org/dp/rogerbot
rogerbot is the name of the SEOmoz crawler bot, the above page has all the info you require.
Regards
Simon
-
The seomoz user agent is named rogerbot. You can read more about the SEOmoz crawl process here: http://seomoz.zendesk.com/entries/20034082-lesson-5-crawl-diagnostics
<code>User-agent: rogerbot Allow: /</code>
-
Thanks Ryan for your immediate reply.
Can you please provide name & the code of the SEOmoz Crawler that I need to enter on my file so that the SEOmoz crawls all the webpages of my website.Apart from SEOmoz Crawler I don't want any other crawler to crawl my website?Please help.Awaiting your reply.
Regards,
Prashakth Kamath
-
That error is pretty straight forward and indicates you have a robots.txt file which is blocking the crawler from accessing your site. The robots.txt file can be read by going to your site URL and adding /robots.txt to it such as www.mysite.com/robots.txt.
The file can be found in the root directory on your site's web server. Remove or alter the file to allow search engines to crawl your site. More info can be found at http://www.robotstxt.org/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can someone kindly explain what 'Crawl Issue Found: No rel="canonical" Tags' means? Is this a critical error and how can it be rectified?
Can someone kindly explain what 'Crawl Issue Found: No rel="canonical" Tags' means? Is this a critical error and how can it be rectified?
Moz Pro | | JoshMcLean0 -
What is error message:social_account.no_method?
When trying to add a twitter account to track, I received the following error message:social_account.no_method - has anyone else received this message?
Moz Pro | | solutionbuilt.com0 -
Looking to pull a report from MOZ with global search volume
I'm in need of a monthly report on campaign keywords with rank AND global search volume (not "just" traffic.) Is this doable from MOZ? More generally, what does monthly SEO reporting to the C-Suite look like for most SEO's? Would be a good "#WhiteboardFriday" topic. We are a mid sized ecommerce retailer.
Moz Pro | | TEMurray0 -
"Duplicate Page Title" Problem - Please Help
Hello, My website is categorized into 2 main categories. Sci/Tech (Has 4 sub-categories) Gadgets(Has 2 sub-categories) The Crawl diagnostic tool shows "Duplicate Page Title" error on Gadget's sub-categories while there's no error on the Sci/Tech. I don't really know how to get rid of these errors. Anyone has a solution to this?
Moz Pro | | MighteeObvious0 -
Getting SEOMoz reports to ignore certain parameters
I want the SEOMoz reports to ignore duplicate content caused by link-specific parameters being added to URLs (same page reachable from different pages, having marker parameters regarding source page added to the URLs). I can get Google and Bing webmaster tools to ignore parameters I specify. I need to get SEOMoz tools to do it also!
Moz Pro | | SEO-Enlighten0 -
SEO Targeting: Ideal monthly keyword search volume?
What's your SEO sweet spot for monthly keyword search volume*? At what volume do you actually expect to see traffic? (I know this depends heavily on competition for said keywords). (*Currently measuring volume via Google AdWords Keyword Tool / SEOmoz keyword difficulty tool. || Comparing exact match and broad match.)
Moz Pro | | JessicaCox0 -
Why can I only add three search engines to a campaign?
I need to track global keywords rankings for a web based product. Only being able to add three search engines to a campaign is limiting. I need to track at least 5 regional versions of Google, and curently have to duplicate my campaign to do this. would prefer to keep it under one campaign.
Moz Pro | | Vend0