The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Blocking internal search results
Hello Everyone, Does anyone know how I can block Google from indexing internal search results? Thanks. Ryan
On-Page Optimization | | RyanUK0 -
Can I robots.txt an entire site to get rid of Duplicate content?
I am in the process of implementing Zendesk and will have two separate Zendesk sites with the same content to serve two separate user groups (for the same product-- B2B and B2C). Zendesk does not allow me the option to changed canonicals (nor meta tags). If I robots.txt one of the Zendesk sites, will that cover me for duplicate content with Google? Is that a good option? Is there a better option. I will also have to change some of the canonicals on my site (mysite.com) to use the zendesk canonicals (zendesk.mysite.com) to avoid duplicate content. Will I lose ranking by changing the established page canonicals on my site go to the new subdomain (only option offered through Zendesk)? Thank you.
On-Page Optimization | | RoxBrock0 -
Too Many on page links! Will "NoFollow" for navigation help?
I am getting to many on page links ( for all my pages). Here is my website: http://www.websterpowerproducts.co.uk I think it is to do with the the navigation bar down the right hand side. I don't really want to get ride of this as it offers users a way of getting where they want without lots of clicking. I was wondering if adding a "NoFollow" tag to each of they links would stop the link juice getting diluted by the navigation bar. Many Thanks
On-Page Optimization | | WebsterPowerTools0 -
Would adding noindex help?
I had completely forgotten that I have about 20 pages of content on my site that is an exact duplicate of other sites (i.e. obtained from PLR site). I really do not want to delete these pages as they do get a lot of visitors (or did before last algo updates). These visitors are not from organic search but have navigated to the pages from within my site. Question is should I a) add noindex to these pages and then ask google to remove them from index or b) try to rewrite them Many Thanks Simon
On-Page Optimization | | spes1230 -
Can embedding videos from other sites help SEO?
I work for an eCommerce site and many of the vendors we get our products from have videos our their site that explain how the products work and the different features of them, ect. We really have done nothing with video marketing, and would like to create some of our own unique videos, but this will be impossible for the vast majority of the products on our site. We know from a content standpoint embedding videos from our vendors to our product pages will be beneficial, but at the same time it shows our competitors and other site visitors where we are getting our products from which we would rather not do. How will this effect us from an SEO standpoint though? Will it help us to have new content, even though it is embedded from another site? Or will it hurt us because it is not unique? Or will it have no effect at all? Thanks for you help!
On-Page Optimization | | ClaytonKendall0 -
Is there a report in SEOMoz that will show me what keywords each page ranks for on my site?
I would like to find all of the keywords not just the keywords that I specified in the tracking section.
On-Page Optimization | | Court_H0 -
What's the best practice for implementing a "content disclaimer" that doesn't block search robots?
Our client needs a content disclaimer on their site. This is a simple "If you agree to these rules then click YES if not click NO" and you're pushed back to the home page. I have this gut feeling that this may cause an upset with the search robots. Any advice? R/ John
On-Page Optimization | | TheNorthernOffice790 -
Are duplicate titles an issue for pages I don't need ranking for?
A client has a load of duplicate page titles on their site. However, to cut a long story short, most of these pages are pointless and therefore we don't need ranking for them. As such, I'm not concerned whether any of the pages with duplicate content on them are ranked or not..... unless having duplicate page titles / content on these pages could mean that other pages on the site, like the homepage, don't rank as high because of this. Do I need to worry about duplicate titles on these pages, or can I ignore duplicate content on pages that I don't want to be ranked? Hope that makes sense!
On-Page Optimization | | RiceMedia0