How to get rid of the message "Search Engine blocked by robots.txt"
-
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest.
Regards,
Prashakth Kamath
-
Thanks Simon for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Thanks Ryan for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Hi Sagar
That was a good reply from Ryan.
Check out http://www.seomoz.org/dp/rogerbot
rogerbot is the name of the SEOmoz crawler bot, the above page has all the info you require.
Regards
Simon
-
The seomoz user agent is named rogerbot. You can read more about the SEOmoz crawl process here: http://seomoz.zendesk.com/entries/20034082-lesson-5-crawl-diagnostics
<code>User-agent: rogerbot Allow: /</code>
-
Thanks Ryan for your immediate reply.
Can you please provide name & the code of the SEOmoz Crawler that I need to enter on my file so that the SEOmoz crawls all the webpages of my website.Apart from SEOmoz Crawler I don't want any other crawler to crawl my website?Please help.Awaiting your reply.
Regards,
Prashakth Kamath
-
That error is pretty straight forward and indicates you have a robots.txt file which is blocking the crawler from accessing your site. The robots.txt file can be read by going to your site URL and adding /robots.txt to it such as www.mysite.com/robots.txt.
The file can be found in the root directory on your site's web server. Remove or alter the file to allow search engines to crawl your site. More info can be found at http://www.robotstxt.org/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
MOZ Not Picking Up Facebook "Likes"
For some reason MOZ is not registering the number of Facebook likes my corporate Facebook page has. Over 700 likes have accumulated, yet MOZ is showing zero. It is strange because MOZ has picked up three Facebook "shares". Does anyone have an idea how this issue can be corrected? Thanks,
Moz Pro | | Kingalan1
Alan0 -
What are the factors that have an impact on the google search results?
When I compare the ranking of my keywords with the Moz Rankings and my own search results I get different results even I use the google chrome incognito browser (not logged in, no cookies, no cache) and it's definitely the same country-setting (google.de) where the search is run. I double checked it with several browsers and computers, and I still get a different result as the Moz Ranking Tracker. The keyword is "schafkopf" and the site is www.sauspiel.de. The relevant country is Germany (www.google.de). Does anyone know the reason why the results differ? Could it be the search query was made from different countries/IPs? Any ideas, hints?
Moz Pro | | sauspiel0 -
Is there any way to get Moz to generate a regular (e.g. weekly) report of new links to my domain, automatically?
I want to know when new people link to my domain. I do not want to log in once a week and use a complex interface to figure this out, I want it in my inbox every week. Is there a way to get Moz to do this? (I am a PRO user) Cheers, Daniel
Moz Pro | | swombat0 -
About how long does it take Google WMT to refresh stats on "Links to Your Site"?
About how long does it take Google WMT to refresh stats on "Links to Your Site"? We're dealing with an unnatural link/anchor phrase issue and I'm curious as to the "typical" time it takes for Google to recognize the links removal or Anchor Text change. Any refresh time ideas on OpenSiteExplorer or AHREFS as well would be a plus... Thanks! Dan Using this guide (very helpful thanks SEOmoz!) http://www.seomoz.org/blog/identifying-link-penalties-in-2012
Moz Pro | | MTteam0 -
"Rank Tracker Tool" is not agreeing with "Keyword Difficulty Tool"
I usually don't sweat a lot for ranks and such but last couple of days, our rankings have been moving drastically. 'Rank Tracker' shows 1st position for many keywords and the "keyword difficulty tool" shows 2nd and 3rd positions. Is is just me or this is a common thing?
Moz Pro | | Syed10 -
How to get seomoz to re-crawl a site?
I had a lot of duplicate content issues and have fixed all the other warnings. I want to check the site again.
Moz Pro | | adamzski0 -
Is there a way to see all SEOMoz questions, comments, posts, responses that I've given a "thumbs up" to?
Lots of times I'll Thumbs Up a question, blog post or response not only if I think it's high quality but if I think I'd like to revisit it in the future. Essentially, I want to use a the SEO Moz Thumbs Up as a bookmark. Is there any way to see all the SEO Moz content that I have given a "Thumbs Up" to? I know I can review all the questions I've asked or answered in the "My Q&A" page. Perhaps, I'm just missing a "Thumbs Up" page somewhere on my profile?
Moz Pro | | TaitLarson2 -
"Issue: Duplicate Page Content " in Crawl Diagnostics - but these pages are noindex
Hello guys, our site is nearly perfect - according to SEOmoz campaign overview. But, it shows me 5200 Errors, more then 2500 Pages with Duplicate Content plus more then 2500 Duplicated Page Titles. All these pages are sites to edit profiles. So I set them "noindex, follow" with meta robots. It works pretty good, these pages aren't indexed in the search engines. But why the SEOmoz tools list them as errors? Is there a good reason for it? Or is this just a little bug with the toolset? The URLs which are listet as duplicated are http://www.rimondo.com/horse-edit/?id=1007 (edit the IDs to see more...) http://www.rimondo.com/movie-edit/?id=10653 (edit the IDs to see more...) The crawling picture is still running, so maybe the errors will be gone away in some time...? Kind regards
Moz Pro | | mdoegel0