Is it hurting my seo ranking if robots.txt is forbidden?
-
robots.txt is forbidden - I have read up on what the robots.txt file does and how to configure it but what about if it is not able to be accessed at all?
-
Yes, excluding certain pages can be a benefit to your rankings: if the excluded pages could be considered duplicate content with your marketing pages or with it each other.
This is usually the case for blogs (think wordpress categories) or webshops (pagination, as well as single product pages reachable by different paths (and thus having different urls). As Ryan pointed out: controll that on the page level via noindex,follow to allow PR to flow. Use noindex,nofollow for "internal" pages you dont want to see crawled.
I am not sure, but having 9950 pages indexed, but considered duplicate content might hurt rankings for other pages on that domain. Google might consider the Domain spammy.
If you need a specific hint for your domain, send me a PM and I have a look if time permits.
-
In general, I do not use robots.txt. It is a better practice to use "noindex" for the pages you do not wish to have indexed.
If I had a 10k page site with 50 marketing pages, I would either want to index the entire site, or question why the other 99% of the site exists if it does not help market the products. There are numerous challenges your scenario prevents. If you block 99% of your site with robots.txt or the noindex meta tag, you are severely disrupting the flow of PR throughout your site. Also you are either blocking content which should be indexed, or you are wasting time and resources creating junk pages on your site.
If the content truly should not be indexed, it likely should be moved to another site. I would need a lot more details about the site, it's purpose and the pages involved. Whatever the proper solution, it is not likely going to be using robots.txt to block 99% of the site.
-
So in regards to increasing ranking, is there a benefit of using the robots.txt file to only index certain "marketing" page and exclude other content that may dilute your site. For example, lets say I have 10,000 pages but only about 50 or so are my marketing page. Would using robots.txt to only crawl my main marketing pages help place emphasis on that content?
-
Sebes is correct. To add a bit more, it is not necessary to provide a robots.txt file. Actually, it is preferable in most cases not to use the file but it is necessary if you do not have direct control over the code used in every page of your site. For example, if you have a CMS or Ecommerce based site you may not have likely do not have control over many pages on your site which are automatically generated through the software. In these cases the only way you can control how crawlers will treat your site's pages is either to pay for custom modifications to your site's code or to use a robots.txt file.
-
If the robots.txt can not be read by google or bing they assume that they can crawl as much as they want to. Check out the google webmaster tool to see whether google can "see" and access your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does Google push down for not ranking top for branded keywords?
Hi all, Usually websites rank for their branded keywords. Some times third party websites takeover the websites for branded keywords. If there are more number of such queries where website is not ranking (top) for branded keywords, Google push down website in overall rankings? Any correlation? Thanks
Algorithm Updates | | vtmoz0 -
SEO Audit after Penguin 2.1 what are you guys seeing? this is my thougts
We have looked at around 2000 sites since Penguin 2.1 launched a few weeks back. These include our customers and their own competitors site. We are going through all the data which is obviously going to take some time. Hopefully we will publish a report on our findings as we are happy to share. What I currently see in my early analysis is Roughly 70% of sites tested have 0% exact match Anchor Text for their money keywords. The other 30% have less than 5% exact match Anchor Text. The quality of the links is often still poor to the sites ranking on page 1. The content surrounding the links is only about 10-15% of the time related to the money keywords. The loading time of the sites ranking seems to not matter, we encountered a lot of slow sites. Design and usability of the site was not important. We are not seeing much impact via Social media, a lot of these sites are small business Less than 10% of sites on page 1 had a Google+ account More than 40% of page 1 sites had Facebook profiles. More than 80% of the sites ranking on page 1 had less than 100 links to the landing page that ranked What are your opinions of helping to recover if hit by the above??? Q) If you have too high an anchor text percentage and have been hit or may get hit in the future would you. a) create some more high quality links with more varied anchor text, ie Click here, brand name etc b) not create any more links just remove the links you have to dilute the anchor text c) change the anchor text on links you are able to These figures are a work in progress so data will change just wanting to share our early findings and try to get a good conversation going. What are you guys seeing?
Algorithm Updates | | tempowebdesign0 -
Recent Rank drop after Penguin 2.1?
Recently, a lot of pages from our website have moved from page one or ranking number one, to page ten or something. We got a manual penalty message from Google Team, we removed a lot of unnatural links pointing to our pages and disavowed the rest. This got the penalty removed and we got a message from Google confirming the same. Before the manual penalty we were getting about 140,000 visits per day, after the penalty about 80,000. However, after Hummingbird or Penguin 2.1 all our ranks have vanished. We are nowhere in Google for our primary keywords and we getting like 40,000 visits per day. Most are direct or from sources other than Google. We had another look at the links we disavowed, a list of about 11000 domains, we found about 3000 domains to be good. We fixed the disavow file about one week back, but no changes in traffic since. We checking the domains again to see if we have missed more good domains in there; yes, we have. There are still a very few good domains in there. But we are not touching the disavow list; waiting to see the change for the last submitted. We have a dedicated user base, good liking on Facebook, all the stats in Analytics speak good, about 40% repeat visits about 30% direct. About 3000 people search for the site using our brand name as reported in Analytics. I doubt the on-page optimization, the pages could be over-optimized. But the on-page factors for other pages ranking for the keywords are similar. The keyword density is similar, so are the usage of headings and stuff. We have not made any recent changes to these on-page patterns. Our team is not able to figure out what could have gone wrong.
Algorithm Updates | | Develop410 -
3 Subdomains - Can their authority or ranking affect each other?
HI, We have 3 different subdomains that target completely different industries.
Algorithm Updates | | danialniazi
a.example.com - school reviews
b.example.com - healthcare reviews
c.example.com - car reviews Q1: If any of the above subdomains gets demoted or penalized by Google would that affect the other subdomains? Q2: Moving forward with all the new changes in SEO, penalties, etc; would it be wise to utilize 3 subdomains for different industries or should we consider getting separate domains?0 -
Any SEO thoughts about Google's new Data Highlighter for products?
After searching around on the web for a while I couldn't find any case studies or interesting posting about Google's new feature to highlight structured data. In Google Webmaster Tools you can now tag your products to be displayed as structured data in Google's search results. Two questions that rose immediately: 1. What effect will Google's new Data Hightlighter for products have on your SEO? Can we expect better CTR's for productspage results in Google? Better conversion rates perhaps? Any case studies that show KPI improvements after using structured data for products? 2. I would love to see some examples in the search results to see what productpages would look like after Data Highlighting it. Your thoughts or input about this subject will be much appreciated.
Algorithm Updates | | SDIM0 -
I can't understand why I am not rank one on SERPS
Hi Guys, I really cannot understand why I am no longer rank 1 on SERPs? My link data shows great weight in comparison to competitors, my on page SEO is good, nice and diverse on the alt text. I know there are a lot of factors that effect SERPs but I believe I have done well but am still not ranking? Have I missed something?
Algorithm Updates | | TomLondon
I really appreciate any thoughts and ideas. Thanks,
Tom0 -
Why some sites doesn't get ranked in Google but in Bing and Yahoo
Few of my sites e.g. Business-Training-Schools.com and Ultrasoundtechnicians.com doesnt get much visits from Google but these sites get top ranked in Bing and Yahoo. I have tried searching for answer to these question but i did not find anything convincing.
Algorithm Updates | | HQP2 -
What are the good strategies using satellite sites in SEO??
Hello to everybody, We'are thinking about launching a massive amount of satellite websites in order to promote our website. Is it really efficient in terms of link building? Or is the ROI really small due to the amount of time and money needed to create and manage these websites? Thanks a lot!!! Update: Thanks to all of you for all these interesting answers!
Algorithm Updates | | sarenausa1
