Is it hurting my seo ranking if robots.txt is forbidden?
-
robots.txt is forbidden - I have read up on what the robots.txt file does and how to configure it but what about if it is not able to be accessed at all?
-
Yes, excluding certain pages can be a benefit to your rankings: if the excluded pages could be considered duplicate content with your marketing pages or with it each other.
This is usually the case for blogs (think wordpress categories) or webshops (pagination, as well as single product pages reachable by different paths (and thus having different urls). As Ryan pointed out: controll that on the page level via noindex,follow to allow PR to flow. Use noindex,nofollow for "internal" pages you dont want to see crawled.
I am not sure, but having 9950 pages indexed, but considered duplicate content might hurt rankings for other pages on that domain. Google might consider the Domain spammy.
If you need a specific hint for your domain, send me a PM and I have a look if time permits.
-
In general, I do not use robots.txt. It is a better practice to use "noindex" for the pages you do not wish to have indexed.
If I had a 10k page site with 50 marketing pages, I would either want to index the entire site, or question why the other 99% of the site exists if it does not help market the products. There are numerous challenges your scenario prevents. If you block 99% of your site with robots.txt or the noindex meta tag, you are severely disrupting the flow of PR throughout your site. Also you are either blocking content which should be indexed, or you are wasting time and resources creating junk pages on your site.
If the content truly should not be indexed, it likely should be moved to another site. I would need a lot more details about the site, it's purpose and the pages involved. Whatever the proper solution, it is not likely going to be using robots.txt to block 99% of the site.
-
So in regards to increasing ranking, is there a benefit of using the robots.txt file to only index certain "marketing" page and exclude other content that may dilute your site. For example, lets say I have 10,000 pages but only about 50 or so are my marketing page. Would using robots.txt to only crawl my main marketing pages help place emphasis on that content?
-
Sebes is correct. To add a bit more, it is not necessary to provide a robots.txt file. Actually, it is preferable in most cases not to use the file but it is necessary if you do not have direct control over the code used in every page of your site. For example, if you have a CMS or Ecommerce based site you may not have likely do not have control over many pages on your site which are automatically generated through the software. In these cases the only way you can control how crawlers will treat your site's pages is either to pay for custom modifications to your site's code or to use a robots.txt file.
-
If the robots.txt can not be read by google or bing they assume that they can crawl as much as they want to. Check out the google webmaster tool to see whether google can "see" and access your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
About porn sites and ranking
Hello, I'm thinking to extend my website into porn. At the moment there is no pornography on it, although we do talk about sex related topics and products (from dating to tutorials, to toys etc.) Would it be dangerous to keep the porn section on the same domain as the rest? Would this negatively affect my non-porn content as Googlebot would "flag" my website as being pornographic (although only a few pages would be)? Or simply Googlebot would leave the current non-porn pages ranking as they are now, just fine, and plus it would rank the porn pages if they "deserve" to? I hope my question is clear. I don't want to create a subdomain.
Algorithm Updates | | fabx0 -
Number or percentage of new visitors impact Google rankings?
Hi all, Does the number/percentage of new visitors (from different IPs and countries) impact the Google rankings? If there are more number of new visitors, will Google favours the website in rankings considering the fact that new visitors are better than returning/same visitors? Thanks
Algorithm Updates | | vtmoz0 -
Should I document rankings from an Incognito Window?
Mozzers, what are your thoughts on documenting SEO results from an incognito window? Does anyone have any articles they can recommend that talk about this subject? Is it better to document results from the normal window or from an incognito window?
Algorithm Updates | | TaylorRHawkins1 -
Delay between being indexed and ranking for new pages.
I've noticed with the last few pages i've built that there's a delay between them being indexed and them actually ranking. Anyone else finding that? And why is it like that? Not much of an issue as they tend to pop up after a week or so, but I am curious. Isaac.
Algorithm Updates | | isaac6630 -
Looking for Search Engine Ranking Factors more recent than 2013
The 2013 Search Engine Ranking Factors study is a very useful study. However, it was completed more than two years ago, and a lot of algorthim updates have been made since then. Is there a more recent study of this than the one produced in 2013? Any and all information would be valuable. I am also trying to understand the importance of site speed as a ranking factor. Thanks.
Algorithm Updates | | JorgeUmana0 -
Half my ranks have gone missing
I have a site that was rankly very highly for gaming walkthrough phrases. I was ranking #3 for the phrase skyward sword walkthrough Then it began to drop, despite me building more links (non spammy links) through guest blogs and emailing people requesting they just link to it from their site. I went in and added more text to the page and it went up a few places but is now gone from the first 5-10 pages. This has also happened to all of my walkthrough related ranks. My other ranks are still there but these accounted for a large portion of my traffic and is very odd. My on-page grade is near if not perfect my page Some insight would be nice
Algorithm Updates | | webfeatseo0 -
How do I rank multiple pages for my busness/domain name?
When someone searches for our business's name (which is also the domain name) we have one listing (with sitelinks) at the top - however I would also like to rank 2nd, 3rd and 4th for this term. Any suggestions on how this might be done? Thanks.
Algorithm Updates | | CaBStudios0 -
How to write a good resourceful SEO enabled article
We have our saas based website - most of our online customers are those who keep coming back to us and my GA is full of their footprints. I completely want to concentrate on getting hold of those who might really need our software and as of now are not able to find them . Including keywords through which people might want to find us is one of the ways. Next how do I publish that to the majority of the users to find and get traction better on that article or post? Would posting links to facebook twitter etc and getting people to find those articles there and link back and come on our main website to read it - will this help? We sell cloud based software but have various domains where our customers can make use of it. There are at least 5-10 of them. We don't have content at all on our website. In a few simple steps how can I get started with this - Content generation **Linking back the content ** Generating good foot falls from users to those cotent Notching up on google for those content page A detailed insight would prove much helpful Thanks
Algorithm Updates | | shanky11