The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Need help with Meta tags generator
Hello, I am doing SEO for my business website. I am trying to generate Meta Tags here
On-Page Optimization | | webdropyyc
https://webdrop.ca/seo-tools/meta-tag-generator What should I enter for "Search engines should revisit this page after ". We are not going to change content on daily basis. 7 Days or 30 days are good enough?0 -
Do we need to worry about internal duplicate content?
Hi, I have a question about internal duplicate content. We have a catalogue of around 4000 products. Most of these do have individual descriptions but for most of the products they contain a generic summary that includes a sentence to begin with that includes each product name. We're currently working on descriptions for each product, but as you can imagine it's quite a chore. I was wondering if there are actually any penalties for this or whether we can ignore the crawl errors from the moz report? Thanks in Advance!
On-Page Optimization | | 10dales0 -
I'm starting an online training school and need some advice.
Hey Moz'ers, I'm looking start an online training website. I feel there is 3 options: Self hosted LMS (Moodle, Litmos etc), Sass solution (Academy of Mine, Pathwright etc) Or DIY with Chargify type payments/subscriptions. To get started it is easiest to use one of the software as a service options. I get to use my own domain but everything else is via the software provider. **What potential problems can you see if/when I decided to do my own self hosted option? Will I be able to redirect the pages to my new site and pass along my accumulated page rank with minimal disruption or am I missing something catastrophic? ** Any advice would be helpful.
On-Page Optimization | | danlovesadobe0 -
WordPress image urls - need a WP maven
We were having a conversation re urls that are indexed for images that are stored in various media plugins in WP. My question for anyone who is an uberWP person is: What is your opinion re best media storage plugins and how these URLs affect pages on a site for ranking, etc. I realize this is broad, but it is driven out of my concern that I cannot touch everything. When I see a url like this: http://www.drumbeatmarketing.net/wp-content/themes/drumbeat2/img/DB-LOGO-White.png I know there is no way with all the sites and clients we handle that I can get it perfect but this just bugs me for some reason. Should I just chill since it (seemingly) affects so little....?
On-Page Optimization | | RobertFisher1 -
Would adding noindex help?
I had completely forgotten that I have about 20 pages of content on my site that is an exact duplicate of other sites (i.e. obtained from PLR site). I really do not want to delete these pages as they do get a lot of visitors (or did before last algo updates). These visitors are not from organic search but have navigated to the pages from within my site. Question is should I a) add noindex to these pages and then ask google to remove them from index or b) try to rewrite them Many Thanks Simon
On-Page Optimization | | spes1230 -
Do I need a unique post meta description
Just wondering what the best practise is for unique meta desriptions on blogs. When I post a blog on my wordpress, clicking the title takes you to an individual page with that blog on it. I understand how important the title tag is on this page but when I create a meta description: a) is it useful? b) Should it be unique or is it ok to copy part of the post and insert that as the meta tag? Thanks
On-Page Optimization | | acs1111 -
How do you block development servers with robots.txt?
When we create client websites the urls are client.oursite.com. Google is indexing theses sites and attaching to our domain. How can we stop it with robots.txt? I've heard you need to have the robots file on both the main site and the dev sites... A code sample would be groovy. Thanks, TR
On-Page Optimization | | DisMedia0
