The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I hide comment reply's from google? Do I need to?
Reason for asking is Moz reports them as URL too long. Should these even be indexed by google if not, how do I hide them? Example URL : https://www.tansleyphotography.co.uk/farnham-castle-wedding-claire-chris/?reply-to=1876 This really doesn't need indexing, as it's just a comment on a blog post. Does it matter?
On-Page Optimization | | paultansley1 -
Does RSS Feed help to rank better in Google?
Hello, I heard RSS Feed helps in ranking. However, I am not sure if I should enable RSS Feed or not. Whenever I publish an article on my site , I see that many other websites have leeched my Feed and get's the same article I written published with a nofollow backlink to my website article. The worst part is that my article doesn't appear in Google search, but the website which copied my article gets ranked in Google. Although the article gets index on google (checked by using site:website.com). Although some articles show up after 24 hours by ranking higher from the sites which copied my article. Any idea what should I do? Thank you
On-Page Optimization | | hakhan2010 -
I'm starting an online training school and need some advice.
Hey Moz'ers, I'm looking start an online training website. I feel there is 3 options: Self hosted LMS (Moodle, Litmos etc), Sass solution (Academy of Mine, Pathwright etc) Or DIY with Chargify type payments/subscriptions. To get started it is easiest to use one of the software as a service options. I get to use my own domain but everything else is via the software provider. **What potential problems can you see if/when I decided to do my own self hosted option? Will I be able to redirect the pages to my new site and pass along my accumulated page rank with minimal disruption or am I missing something catastrophic? ** Any advice would be helpful.
On-Page Optimization | | danlovesadobe0 -
Magento Canonical & Default Robots Settings
Hello! I'm working with Magento 1.9 for an eCommerce site with several hundred products. Currently I understand it is best practices to use the Canonical tag, however I also have my default robots set to "Index, Follow". Will this cause an issue having product pages set to index, follow but also having a canonical tag included? What are some best practices regarding Magento default robots & canonical tags? Any help is appreciated.
On-Page Optimization | | BretDarby0 -
Does name of town in title tag help if queries don't include the town name?
Hi. Wanted to know if targeting local traffic online and the search volume of KWs in the area do not include the local names (according to KW planner) does it still help to keep the town names in the title tag? does google deliver local results based on location names in title tag if query didn't mention it?
On-Page Optimization | | Morris770 -
How to leverage user reviews & ratings to help in SEO rankings
Folks, We have User Generated Content site in travel domain, we want to leverage reviews & ratings for SEO rankings/CTR . We already are getting micro formats in google but not in all the cases. Any ideas/suggestions will be really very helpful. Thanks in Advance. -Amit
On-Page Optimization | | holidayiq0 -
New CMS system - 100,000 old urls - use robots.txt to block?
Hello. My website has recently switched to a new CMS system. Over the last 10 years or so, we've used 3 different CMS systems on our current domain. As expected, this has resulted in lots of urls. Up until this most recent iteration, we were unable to 301 redirect or use any page-level indexation techniques like rel 'canonical' Using SEOmoz's tools and GWMT, I've been able to locate and redirect all pertinent, page-rank bearing, "older" urls to their new counterparts..however, according to Google Webmaster tools 'Not Found' report, there are literally over 100,000 additional urls out there it's trying to find. My question is, is there an advantage to using robots.txt to stop search engines from looking for some of these older directories? Currently, we allow everything - only using page level robots tags to disallow where necessary. Thanks!
On-Page Optimization | | Blenny0 -
Why isn't SEOMoz using File Extensions (*.html etc) on any of their web page URLs?
...and what is the SEO benefit of this? This video from Matt Cutts suggests using file extentions, except for a directory.
On-Page Optimization | | magicrob0