The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Need Some Quality Vs. Quantity SEO Advice
We have a gallery here with our main categories of patches. https://www.stadriemblems.com/gallery/ If you click on one, say Fire Patches, you'll be taken to a page of just fire patches. https://www.stadriemblems.com/fire-patches/ But here's the kicker: If you notice of the fire patch page, there are also sub-categories to that. So if you click on say, Fire Rescue, you get taken one level deeper. https://www.stadriemblems.com/fire-patches/fire-rescue-patches/ I'm redoing this entire site (a project over five years overdue), and I'm wondering if it's really worth it to keep these three-level deep sub pages. I originally created them with long tail SEO in mind, making us be the only ones who come up when people search for very specific patches. But it's a big undertaking to redo all of them, and are they really adding any value?
On-Page Optimization | | UnderRugSwept0 -
I'm starting an online training school and need some advice.
Hey Moz'ers, I'm looking start an online training website. I feel there is 3 options: Self hosted LMS (Moodle, Litmos etc), Sass solution (Academy of Mine, Pathwright etc) Or DIY with Chargify type payments/subscriptions. To get started it is easiest to use one of the software as a service options. I get to use my own domain but everything else is via the software provider. **What potential problems can you see if/when I decided to do my own self hosted option? Will I be able to redirect the pages to my new site and pass along my accumulated page rank with minimal disruption or am I missing something catastrophic? ** Any advice would be helpful.
On-Page Optimization | | danlovesadobe0 -
Help With Duplicated Content
Hi Moz Community, I am having some issue's with duplicated content, i recently removed the .html from all of our links and moz has reported it as being duplicated. I have been reading up about Canonicalization and would to verify some details, when using the canonical tag would it be placed in the /mywebpage.html or /mywebpage file? I am having a hard time to sort this out so any help from you SEO experts would be great 🙂 I have also updated my htaccess file with the following Thanks in advance
On-Page Optimization | | finelinewebsolutions0 -
Robots file include sitemap
Hello, I see that google, facebook and moz... have robots.txt include sitemap at the footer.
On-Page Optimization | | JohnHuynh
Eg: http://www.google.com.vn/robots.txt Sitemap: http://www.google.com/sitemaps_webmasters.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml Should I include my sitemap file (sitemap.xml) at the footer of robots.txt and why should do this? Thanks,0 -
The need of two-keyword optimization in the same page
Hi there! Due to the business model of my company I have to optimize two keywords in one page. I just asked about this question before and someone told me thant as long as they refer to the same concept and have almost the same "meaning", it is possible. The problem is how to face it up. I mean, there's one H1 label, one title, etc....and what's the "policy" of key distribution in the content in order to priorice these keywords? What you guys recommend? many thanks
On-Page Optimization | | juanmiguelcr0 -
Duplicate content list by SEOMOZ
Hi Friends, I am seeing lot of duplicate (about 10%) from the crawl report of SEOMOZ. The report says, "Duplicate Page Content" But the urls it listed have different title, different url and also different content. I am not sure how to fix this issue.. My site has both Indian cinema news and photo gallery. The problme mainly coming in photo gallery posts. for example: this is the main url of a post. apgossips.com/2012/12/18/telugu-actress-poonam-kaur-photos . But in this post, each image is a link to its enlarged images (default wordpress). The problem is coming with each individual image with in this post. examples of SEOMOZ report 3 individual urls as duplicate content...from the same above post.: http://apgossips.com/2012/12/18/telugu-actress-poonam-kaur-photos/poonam-kaur-hot-photo-shoot-stills-4 http://apgossips.com/2012/12/18/telugu-actress-poonam-kaur-photos/poonam-kaur-hot-photo-shoot-stills-3 http://apgossips.com/2012/12/18/telugu-actress-poonam-kaur-photos/poonam-kaur-hot-photo-shoot-stills-2 Some body please advise me.. Appreciate your help.
On-Page Optimization | | ksnath0 -
Advice needed about site
I would like to get advice about the site http://www.wb-3d.com regarding what we should do to get it highly on Google. This seems to be a tough one as there is not enough content on the site. I would immensly appeciate any suggestion to improve the rankings. Specifically, what we should tweet and update on facebook. Thanks
On-Page Optimization | | seoug_20050 -
Are duplicate titles an issue for pages I don't need ranking for?
A client has a load of duplicate page titles on their site. However, to cut a long story short, most of these pages are pointless and therefore we don't need ranking for them. As such, I'm not concerned whether any of the pages with duplicate content on them are ranked or not..... unless having duplicate page titles / content on these pages could mean that other pages on the site, like the homepage, don't rank as high because of this. Do I need to worry about duplicate titles on these pages, or can I ignore duplicate content on pages that I don't want to be ranked? Hope that makes sense!
On-Page Optimization | | RiceMedia0
