How to get rid of the message "Search Engine blocked by robots.txt"
-
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest.
Regards,
Prashakth Kamath
-
Thanks Simon for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Thanks Ryan for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Hi Sagar
That was a good reply from Ryan.
Check out http://www.seomoz.org/dp/rogerbot
rogerbot is the name of the SEOmoz crawler bot, the above page has all the info you require.
Regards
Simon
-
The seomoz user agent is named rogerbot. You can read more about the SEOmoz crawl process here: http://seomoz.zendesk.com/entries/20034082-lesson-5-crawl-diagnostics
<code>User-agent: rogerbot Allow: /</code>
-
Thanks Ryan for your immediate reply.
Can you please provide name & the code of the SEOmoz Crawler that I need to enter on my file so that the SEOmoz crawls all the webpages of my website.Apart from SEOmoz Crawler I don't want any other crawler to crawl my website?Please help.Awaiting your reply.
Regards,
Prashakth Kamath
-
That error is pretty straight forward and indicates you have a robots.txt file which is blocking the crawler from accessing your site. The robots.txt file can be read by going to your site URL and adding /robots.txt to it such as www.mysite.com/robots.txt.
The file can be found in the root directory on your site's web server. Remove or alter the file to allow search engines to crawl your site. More info can be found at http://www.robotstxt.org/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What's the best way to search keywords for Youtube using Moz Keyword explorer?
I want to optimize my youtube channel using identified keywords, but I'm concerned that the keywords I'm identifying work well for SERP's but might not be how people search in Youtube. How do a distinguish my keywords to be targeted for Youtube?
Moz Pro | | Dustless0 -
Discrepancy between backlinks (opensiteexplorer vs. search console)
Hey there, when I check in search console for "Links to your site" and "your most linked content" I see incoming links pointing to the wrong (country and language) subdirectory? When I check those same backlinks with opensiteexplorer they seem to point (after redirects) to the correct subdirectory. Anyone out there with similar experiences or thoughts on this? I would be happy to give access to search console if you wanted to have a look at this. Cheers, Jochen When I check in search console for "Links to your site" and "your most linked content" I see incoming links pointing to the wrong (country and language) subdirectory? When I check those same backlinks with opensiteexplorer they seem to point (after redirects) to the correct subdirectory. Anyone out there with similar experiences or thoughts on this? I would be happy to give access to search console if you wanted to have a look at this.
Moz Pro | | Online-Marketing-Guy
Cheers,
Jochen0 -
"Loading" error in Open Site Explorer
The "top pages" keeps hanging while trying to retrieve social metrics "FB Shares" "Tweets" and "Google +1". I've tried in Chrome and Firefox. Can I just turn these off? ose-hanging.jpg
Moz Pro | | cpaddock0 -
Getting your site totally indexed by SEOMOZ
Hi guys! Ijust started using SEOMOZ software and wondered how it could be that my site has over 10.000 pages but in the Pro Dashboard it only indexed about 1500 of them. I've been waiting a few weeks now but the number has been stable ever since. Is there a way to get the whole site indexed by SEOMoz software? Thanks for your answers!
Moz Pro | | ssiebn70 -
Duplicate content due to "Email a Friend" and "PhotoGallery"
Crawl Diagnostics gives me 1650 duplicate page content errors. 800+ are for photo gallery, the link upon which my large image is shown. 800+ are for the Email a Friend form page. Presumably SEOMoz's tool is getting there by following the link inside a product page, which as you can see will differ only by the ProductCode of every product.
Moz Pro | | jotham2
www.completemobilehomesupply.com/PhotoGallery.asp?ProductCode=anchor101
www.completemobilehomesupply.com/EmailaFriend.asp?ProductCode=Shutter001 1 - Is it critical to fix?
I am assuming Yes, but if for some reason the answer is No, please share. 2 - Any idea on how to fix?
The site uses Volusion, fyi, so it may be a limiting factor of the platform. Thank you for your time.
Also if this topic has been previously covered, please link and I'll read there instead.0 -
Getting PA & DA off of a list of links
I have a list of links that I want to get PA and DA for each individual link, can this be done in some way other than one at a time? I've heard this can be done with excel and using api but I don't know the specifics of this.. Help would be appreciated
Moz Pro | | Fergclaw2 -
How long should "Term Extractor" be let alone to complete??
It's been left alone to grind now for a good 20 mins, and ah... it's still grinding away. Is it worth just trying again? Or does that typically mean it probably isn't going to work on the domain it's, ah, grinding away on? Thanks
Moz Pro | | iansears0 -
Why is Roger crawling pages that are disallowed in my robots.txt file?
I have specified the following in my robots.txt file: Disallow: /catalog/product_compare/ Yet Roger is crawling these pages = 1,357 errors. Is this a bug or am I missing something in my robots.txt file? Here's one of the URLs that Roger pulled: <colgroup><col width="312"></colgroup>
Moz Pro | | MeltButterySpread
| example.com/catalog/product_compare/add/product/19241/uenc/aHR0cDovL2ZyZXNocHJvZHVjZWNsb3RoZXMuY29tL3RvcHMvYWxsLXRvcHM_cD02/ Please let me know if my problem is in robots.txt or if Roger spaced this one. Thanks! |0