How to get rid of the message "Search Engine blocked by robots.txt"
-
During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest.
Regards,
Prashakth Kamath
-
Thanks Simon for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Thanks Ryan for the info.Will check and revert back if there is any issues.
Regards,
Prashakth Kamath
-
Hi Sagar
That was a good reply from Ryan.
Check out http://www.seomoz.org/dp/rogerbot
rogerbot is the name of the SEOmoz crawler bot, the above page has all the info you require.
Regards
Simon
-
The seomoz user agent is named rogerbot. You can read more about the SEOmoz crawl process here: http://seomoz.zendesk.com/entries/20034082-lesson-5-crawl-diagnostics
<code>User-agent: rogerbot Allow: /</code>
-
Thanks Ryan for your immediate reply.
Can you please provide name & the code of the SEOmoz Crawler that I need to enter on my file so that the SEOmoz crawls all the webpages of my website.Apart from SEOmoz Crawler I don't want any other crawler to crawl my website?Please help.Awaiting your reply.
Regards,
Prashakth Kamath
-
That error is pretty straight forward and indicates you have a robots.txt file which is blocking the crawler from accessing your site. The robots.txt file can be read by going to your site URL and adding /robots.txt to it such as www.mysite.com/robots.txt.
The file can be found in the root directory on your site's web server. Remove or alter the file to allow search engines to crawl your site. More info can be found at http://www.robotstxt.org/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Competitor getting External Links from search.aol.com
Recently, I noticed that one of the competitors I track within my Moz campaign received about 12 new inbound links. As a result, there DA jumped about 10 points. I reviewed these new external links and was surprised to see that they are all "search.aol.com/aol/search?query= ..." with Link Anchor Text that is good for the industry we compete in. Can anyone tell me why these are being counted as "Inbound Links". It just doesn't seem right. Is this some sort of black hat seo tactic?
Moz Pro | | itvisionsinc0 -
8 point decrease in DA, increase in search traffic- should I disavow links?
Would a decrease in DA reflect the need to disavow junk links? My branded keywords recently bumped down two spots, whereas the keywords which have been bringing me the most traffic have been unaffected. Site: Edwardsturm.com DA decrease of 8 over past 3 months. Keyword "Edward Sturm" recently moved to second spot.
Moz Pro | | Edward_Sturm0 -
Discrepancy between backlinks (opensiteexplorer vs. search console)
Hey there, when I check in search console for "Links to your site" and "your most linked content" I see incoming links pointing to the wrong (country and language) subdirectory? When I check those same backlinks with opensiteexplorer they seem to point (after redirects) to the correct subdirectory. Anyone out there with similar experiences or thoughts on this? I would be happy to give access to search console if you wanted to have a look at this. Cheers, Jochen When I check in search console for "Links to your site" and "your most linked content" I see incoming links pointing to the wrong (country and language) subdirectory? When I check those same backlinks with opensiteexplorer they seem to point (after redirects) to the correct subdirectory. Anyone out there with similar experiences or thoughts on this? I would be happy to give access to search console if you wanted to have a look at this.
Moz Pro | | Online-Marketing-Guy
Cheers,
Jochen0 -
My website not getting organic hits
Hello Moz, I am enjoying being here i just bought moz pro, and am digging for what to change in my website, Apart from that I just wanna know why my website not getting rank in search engine, if someone can do quick audit for me it will be obligation on me. querease(dot)com
Moz Pro | | querease0 -
What is Linking C-Blocks
Currently i am using MOZ pro tool under moz analyticls >> Moz Competitive Link Metrics >> history having a graph "Linking C-Blocks" Please help me understanding Linking C-Blocks, what is, How to build, how to define ...
Moz Pro | | shankar3334 -
How could I improve my "brand" and "mentions" using the Moz tool?
Hello I am a bit stuck on what to put into this section of MOZ. As you can see from the attached image I have setup to catch any mention of my business name and root url. But beyond that I am uncertain of what to use to monitor who is talking about us. I'm also uncertain of what other abilities or usefulness this function would have. aOq7wVg.png
Moz Pro | | infinart0 -
How do you run ranks now for clients that aren't in your local area now that Google customizes search results based on location?
How do you run ranks now for clients that aren't in your local area now that Google customizes search results based on location? What tool/service have you found that gives accurate rank results? To me it seems like there is no guaranteed global #1position any more, or is this the situation for only certain types of searches? Likewise how can anyone tool give a global accurate rank score?
Moz Pro | | greengroup0 -
What should be cols value if I want to get Backlinks?
Hi, I am forming below url to get backlinks. http://lsapi.seomoz.com/linkscape/url-metrics/".$trimurl."?Cols=2048 &AccessID=".$accessID." &Expires=".$expires." &Signature=".$urlSafeSignature; For Example, if I keep $trimurl = "www.tatvic.com/" , I get [uid] => 633. Is this a right way to get number of backlinks ? If not, what should be the 'Cols' value? Also, how can I ensure that the number of links I am getting is correct ? Is there any way to compare this number with Google search results? This is very essential to check as I got different number of backlinks on different APIs. Thank you.
Moz Pro | | Ravi_Pathak0