The SEOmoz crawler is being blocked by robots.txt need help
-
SEO moz is showing me that the robot.txt is blocking content on my site
-
Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.
A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.
You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.
-
Hi Jason,
What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.
-
Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.
You can use Google Webmaster tools to help you make a proper robots.txt file.
Best of luck
-
Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt
-
what needs to be done in the htaccess file. ? can anyone give me a step by step process
-
I would look at your htaccess file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are there detrimental effects of having multiple robot tags
Hi All, I came across some pages on our site that have multiple robot tags, but they have the same directives. Two are identical while one is for Google only. I know there aren't any real benefits from having it set up this way, but are there any detrimental effects such as slowing down the bots crawling these pages? name="googlebot" content="index, follow, noodp"/> Thanks!
On-Page Optimization | | STP_SEO0 -
I need a fresh eye on this project, as I ran out of ideas
I know generally it's not a good idea posting project urls, however this time I'll break the rule as I could really use some genuine ideas: I just can't explain my rankings. So I created a small site, branded term, english language, I will use my British ranking data, you can use justproxy.co.uk to verify if you'd like. Site is 2 years old. My main keyword is Breast Actives, however my I describe the strangest phenomenon after the bullets. Site: http://buybreastactives.info So the point is: I did a little keyword stuffing back in the days, but removed it, took a safe middle density (analyzed the first rankers keyword usage). I've been looking at competitors backlink profiles comparing them to mine. The growth rates are similar. I have a little lower Domain and Page authority than the first few sites, but not that lower. Backlink quantity is somewhat similar to first 20, yet still I am ranking somewhere way beyond 200. I am not expecting a first place with a "somewhat similar" profile, but my bet is I should be at least in top 50. The only technique that worked was regular posting, I brought up the site into top 70, but as the site is really focused on a particular product, and the site's brewity suffered the price of posting semi-related articles, so I stopped. Rank out beyond 200. Another intresting aspect I cannot get my head around is that for "breast actives reviews" I have a dedicated page buybreastactives.info/real-customer-reviews, with some keyword usage, and another page buybreastactives.info/worldwide is ranking for it (around 80-90), but it does not use this keyword at all. So if you have some time to give it a glance I would be greatful for some thoughts. EDIT: Before penguin around 2012 dec site was in top 10 for nearly all terms targetted, with the stuffed keywords.
On-Page Optimization | | snetface0 -
E-commerce site hit by the latest Panda, please help
Hello mates, Our site is an e-comerce site http://goo.gl/4QBVjR has been hit hard by the latest Panda on the last weekend as many other sites such as Retailmenot etc. We were working hard from one year til now:1. Rewrite all the pages with pour content (less than 100 words), and non-original content.2. The pages where was impossible rewrite the content were 410 gone.3. About the link profile, we were removing all the unnatural links by contacting thewebmaster and finally updating a DISAVOW file at monthly basis.All these tasks were performed and we see some improvements close to the original traffic before 2012Now from the last weekend our traffic down again, so we need to know if we can do anythingor there are something hidden from our eyes.Thank youClaudio
On-Page Optimization | | SharewarePros0 -
Our urls for adwords are slightly different from current urls presented on site (weused htaccess to help create shorter urls). How important is it that the adwords url match the sitemap url for keywords on those pages?
Hello, We have dynamic urls that we have made into short urls through htaccess and code manipulation. Some of our adwords urls are different from our page urls - for example a) Latest version of page www.abc.com/x-y-z.html b) Previous version of url www.abc.com/x+y+z.html c) raw original version www.abc.com/yyy/zzz?category=X&Product-code=Y etc etc. Would my ranking for keywords on the page improve if I diligently made all of them the same? They all go to the same page even now, and no 404 errors or anything. Thanks Sam
On-Page Optimization | | samgold0 -
SEOmoz Says 530 - Google Webmaster Tools Says 16
A SEOmoz crawl report on 1/20/2013 says we have 530 duplicate page titles most of which are pages like this these two: http://www.audiobooksonline.com/reviews/review.php/details/38 http://www.audiobooksonline.com/reviews/review.php/details/104 These pages are the result of allowing visitors to post their opinions and ratings about specific audio books. Google Webmaster Tools reports we have 16 duplicate page titles none of which are a review page. Why is SEOmoz reporting duplicate page titles? Should SEOmoz be reporting these pages as having duplicate page titles? Why isn't Google Webmaster Tools reporting the review pages as having duplicate titles?
On-Page Optimization | | lbohen0 -
Large block of several thousand words on homepage of Ecommerce site - opinions?
I work for a local SEO company who are continually adding content to the home page of clients websites. While i do agree that it is a good idea to add content to the home page and to link to inner pages, the home page of several clients exceeds 6,000 or more words. Every month, an article based on a brand or category (which already has its own page which is optimised with an article) is added to the home page with links to inner pages. I have stressed that it is not a good idea to have so much content on the home page, and even more so that it shouldn't target all the same keywords. I have pointed out that many of the brand and category pages do not rank well for their keywords, whereas in most cases it is the home page which is ranking instead, which i have suggested this is because the home page is too well optimised for those keywords. What are your opinions on this? Are you for or against continually adding content (which already has its own designated page) to the home page of an Ecommerce website? I should also add that this content is within a drop down div box at the footer on some sites and just above the footer on other sites. The category and brand pages also have drop down divs with an article just below the header.
On-Page Optimization | | Kinsel0 -
Rel Canonical - Could someone please help confirm something?
Morning Mozzers, I'm looking at a site (www.zitan.co.uk) and making a few recommendations for SEO, one of the things I've spotted is something weird with rel canonical. It looks (to me) as if they've got almost every single page set with this tag: rel="canonical" href="http://www.zitan.co.uk" /> I'm 99% certain that this means that every page on the site (that has this tag) is pointing all link juice / authority back to the homepage? If someone could please check and just confirm that, I'd really appreciate it. Thanks in advance, James
On-Page Optimization | | JamesMio0 -
How to use good keyword URL to help main site
Hi. I'm a long time ecommerce guy and starting a third business. The main site URL is the name of the new business but I also purchased a .com URL that is our #1 keyword to target. So I need to know the best strategy to use the keyword url for helping with getting a top ranking for that keyword. I'm curious if I can or should build out the keyword URL site for the search engines and use a 301 redirect. Can you get top ranking for a site that just redirects? Anyway, I guess you get my question. This keyword gets a ton of perfectly targeted traffic so seems like a goldmine if I work it right. Thanks very much.
On-Page Optimization | | jimmyseo1