How to get rid of the message "Search Engine blocked by robots.txt"
-
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest.
Regards,
Prashakth Kamath
-
Thanks Simon for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Thanks Ryan for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Hi Sagar
That was a good reply from Ryan.
Check out http://www.seomoz.org/dp/rogerbot
rogerbot is the name of the SEOmoz crawler bot, the above page has all the info you require.
Regards
Simon
-
The seomoz user agent is named rogerbot. You can read more about the SEOmoz crawl process here: http://seomoz.zendesk.com/entries/20034082-lesson-5-crawl-diagnostics
<code>User-agent: rogerbot Allow: /</code>
-
Thanks Ryan for your immediate reply.
Can you please provide name & the code of the SEOmoz Crawler that I need to enter on my file so that the SEOmoz crawls all the webpages of my website.Apart from SEOmoz Crawler I don't want any other crawler to crawl my website?Please help.Awaiting your reply.
Regards,
Prashakth Kamath
-
That error is pretty straight forward and indicates you have a robots.txt file which is blocking the crawler from accessing your site. The robots.txt file can be read by going to your site URL and adding /robots.txt to it such as www.mysite.com/robots.txt.
The file can be found in the root directory on your site's web server. Remove or alter the file to allow search engines to crawl your site. More info can be found at http://www.robotstxt.org/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why does moz give different page authority to the same page if a visit comes from adwords vs organic search?
When clicking on an adwords ad the page the landing page has a page authority of 26. When clicking on organic search to the same exact landing page the page authority is 37. Why is this. Does moz or, more importantly Google see these as the same or separate pages? Thanks Tom
Moz Pro | | ffctas1 -
Magento Site "Title Missing or Empty"
Hi everyone . . ..bear with me as I am "noob". My moz analysis brought up a few "critical" issues, one of which was a missing or empty title for a page. That page is: http://www.thirdcoastsigns.com/sales/guest/form Which does not even appear to be a working page . . . so I'm at a bit of a loss ow to fix this. I suspect it's a little bit of SEO expertise and a little bit of Magento expertise. Thoughts and suggestions welcome! P.S. Look forward to my NEXT noob question about a bunch of pages I have flagged as "Pages with temporary redirects". Stay tuned!
Moz Pro | | damon12120 -
Rel=canonical "redirects" to double links
Our devs have set up rel=canonical on our website. First they used relative links href="/dir1/dir2/dir3" for the page http://www.mysite.com/dir1/dir2/dir3/?detail1=1?detail2=2 meaning that it will redirect to http://www.mysite.com/dir1/dir2/dir3, but no luck, the MOZ dashboard showed the tag value to be http://www.mysite.com/dir1/dir2/dir3/dir1/dir2/dir3, then we have decided to rewrite the code, and now the canonical to http://wwwmysite.com/dir1/dir2/dir3/?detail1=1?detail2=2 looks like href="http://www.mysite.com/dir1/dir2/dir3/" but the tag on MOZ looks like http://www.mysite.com/dir1/dir2/dir3http://www.mysite.com/dir1/dir2/dir3. So what is the problem? I really got a problem or MOZ does? The code on website looks exactly like href="http://www.aaa.com/en/bbb/ccc/vvv/nnn/" rel="canonical" /> for the page http://www.aaa.com/en/bbb/ccc/vvv/nnn/
Moz Pro | | apartmentGin0 -
I did a redirect and now I'm getting duplication errors.
I was told by SEO Moz to do a redirect so that our website would be crawled with and without the www in front of the address. I did and now I'm getting duplicate page and title errors because the crawler is seeing www.oursitename.com and its underpages and oursitename.com and its underpages and giving me duplicate page content errors and duplicate page title errors. Makes sense, but how do I make it stop? Anyone else have this problem?
Moz Pro | | THMCC0 -
Canonical URLs for Search Parameters
Hi Guys Our seomoz campaign report is returning a lot or Rel Canonical issues similar to this for each page. The non / version redirects to the / version but how do I get the ones with search parameters ie '?datefrom&nights' to redirect. http://www.lamangaclubresort.co.uk/accommodations/las-brisas-78
Moz Pro | | JohnTulley
http://www.lamangaclubresort.co.uk/accommodations/las-brisas-78/
http://www.lamangaclubresort.co.uk/accommodations/las-brisas-78/?datefrom&nights
http://www.lamangaclubresort.co.uk/accommodations/las-brisas-78/?datefrom=&nights= Any help would be welcome, thanks0 -
Will canonical tag get rid of duplicate page title errors?
I have a directory on my website, paginated in groups of 10. On page 2 of the results, the title tag is the same as the first page, as it is on the 3rd page and so on. This is giving me duplicate page title errors. If i use rel=canonical tags on the subsequent pages and href the first page of my results, will my duplicate page title warnings go away? thanks.
Moz Pro | | fourthdimensioninc0 -
"no urls with duplicate content to report"
Hi there, i am trying to clean up some duplicate content issues on a website. The crawl diagnostics says that one of the pages has 8 other URLS with the same content. When i click on the number "8" to see the pages with duplicate content, i get to a page that says "no urls with duplicate content to report". Why is this happening? How do i fix it?
Moz Pro | | fourthdimensioninc0 -
Can you help me get started using the crawl diagnostics report?
After getting the crawl diagnostics report for the first time my boss and I looked over it and we have tried to fix the problems but we are stumped.I have tried and watched videos , read books, etc.. but have found nothing to help. I need assistance getting started on improving my website. Can you help?
Moz Pro | | WVInjuryLawyer0