Is it hurting my seo ranking if robots.txt is forbidden?
-
robots.txt is forbidden - I have read up on what the robots.txt file does and how to configure it but what about if it is not able to be accessed at all?
-
Yes, excluding certain pages can be a benefit to your rankings: if the excluded pages could be considered duplicate content with your marketing pages or with it each other.
This is usually the case for blogs (think wordpress categories) or webshops (pagination, as well as single product pages reachable by different paths (and thus having different urls). As Ryan pointed out: controll that on the page level via noindex,follow to allow PR to flow. Use noindex,nofollow for "internal" pages you dont want to see crawled.
I am not sure, but having 9950 pages indexed, but considered duplicate content might hurt rankings for other pages on that domain. Google might consider the Domain spammy.
If you need a specific hint for your domain, send me a PM and I have a look if time permits.
-
In general, I do not use robots.txt. It is a better practice to use "noindex" for the pages you do not wish to have indexed.
If I had a 10k page site with 50 marketing pages, I would either want to index the entire site, or question why the other 99% of the site exists if it does not help market the products. There are numerous challenges your scenario prevents. If you block 99% of your site with robots.txt or the noindex meta tag, you are severely disrupting the flow of PR throughout your site. Also you are either blocking content which should be indexed, or you are wasting time and resources creating junk pages on your site.
If the content truly should not be indexed, it likely should be moved to another site. I would need a lot more details about the site, it's purpose and the pages involved. Whatever the proper solution, it is not likely going to be using robots.txt to block 99% of the site.
-
So in regards to increasing ranking, is there a benefit of using the robots.txt file to only index certain "marketing" page and exclude other content that may dilute your site. For example, lets say I have 10,000 pages but only about 50 or so are my marketing page. Would using robots.txt to only crawl my main marketing pages help place emphasis on that content?
-
Sebes is correct. To add a bit more, it is not necessary to provide a robots.txt file. Actually, it is preferable in most cases not to use the file but it is necessary if you do not have direct control over the code used in every page of your site. For example, if you have a CMS or Ecommerce based site you may not have likely do not have control over many pages on your site which are automatically generated through the software. In these cases the only way you can control how crawlers will treat your site's pages is either to pay for custom modifications to your site's code or to use a robots.txt file.
-
If the robots.txt can not be read by google or bing they assume that they can crawl as much as they want to. Check out the google webmaster tool to see whether google can "see" and access your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ranking gone for the original page and a shortened url ranks instead.
Hi Experts!!! Wishing you all a Merry Christmas and Happy New Year In Advance. I am been facing a issue with a few of my SERP results for "Singapore Visa" and related keyword. Until last to last Saturday i.e 16th December, I ranked for Singapore visa keyword with this url https://in.musafir.com/Visa/singapore-visa.aspx !!! But since 18th December I am ranking for "Singapore Visa" keyword with this url and message below it in place of description. Singapore visa - Musafir.com
Algorithm Updates | | sainath
go.musafir.com/Singapore-visa
No information is available for this page.
Learn why The go.musafir.com/Singapore-visa redirects to https://in.musafir.com/Visa/singapore-visa.aspx with some UTM parameters. The URL go.musafir.com/Singapore-visa is a shortened URL which was used for SMS marketing and all of a sudden Google has picked it in SERP instead of Singapore VIsa Landing Page. The Singapore visa Main page is not blocked by Robots.txt file. Please help me to resolve this.1 -
Backlinks to internal pages help website to rank better or vice versa?
Hi Moz community, We have our backlinks mostly pointed to our homepage. We are trying to rank better and not having minimum number of backlinks to our internal pages is one of the things I worry about. Backlinks to homepage alone help in ranking internal pages or backlinks to internal pages help in ranking homepage? Or both required? Thanks
Algorithm Updates | | vtmoz0 -
How soon before author rank becomes a major ranking factor?
Hi, I wanted to pose a question How soon do guys think itll be before author rank becomes a one of Googles major ranking factors? From what I can see the way they have designed it signals that it is only matter of time, before they start using it as a major ranking factor... And I have a question on Author ranks impact on the ability to sell a blog/site in the future. Surely if the blog is tied to an author(s) and the ranking of the site in the search engine is somewhat based on this authors author rank who is a part of the site/blog, then it becomes harder to sell the property if the author is not going to be a part of the property after the sale?. I look forward to your responses on this,
Algorithm Updates | | sanj50501 -
Why are Google Webmaster Tools' Google rankings different to actual Google rankings?
Dear Moz, We have noticed that according to Google Webmaster Tools one of our client sites is ranking very prominently for some of the major key phrases that we are trying to rank them for. However, when we perform a Google search for these queries, our client's content is nowhere to be seen, not even on the 5th page (we logged out of the Google account before performing the test). A long-term manual spam action on our client's site was recently lifted by Google - is it possible that Google Webmaster Tools is providing data about our client's estimated Google rankings, without taking into consideration the penalty of the manual spam action which was taken? Thanks
Algorithm Updates | | BoomDialogue690 -
How does this site rank no 1 for big terms with no optimisation?
Hi, A client recently asked me abut a site that appears to have popped up out of nowhere and is ranking for big terms within their industry: http://bit.ly/11jcpky I have looked at the site for a particular term: Cheap Beds I was using unpersonalised search on google.co.uk with location set to London. The site currently ranks no 1 for that term and other similar terms. The question is how? SEO Moz reports no backlinks (they must have blocked?) Ahrefs and Majestic report report some backlinks but not many and no anchor text with the term in. The Page title and meta do not contain the term nor does the page seem to contain the term anywhere. The domain does have some age though has no keyword match in the URL. I'm a little stumped to how they are achieving these results. Any Ideas Anyone?
Algorithm Updates | | JeusuDigital0 -
SEO Strategy Audit
Greetings Mozzers. I would really appreciate some input / advice / a sanity check on our existing SEO strategy. A year into implementing the strategy (outlined below) there appears to be no uplift in our existing traffic volume. I guess i'm wondering. Am I on the right track? Is there something I'm perhaps additional should be doing / i'm doing something wrong? Am I suffering from Penguin? The strategy: We produce a range of content, this is written 100% for our audience, How-to guides, infographics, industry news etc. - this has lead to getting a wide range of publications with high powered relevant domains, widely shared and read. As a result we now have a series of ongoing columns where we regularly produce articles that our widely read and shared. We have a wide variety of homepage and deep links from quality sources. We have our own blog producing a wide range of content relevant to our audience. Industry trends, news, changes that affect our readers, guides. We optimise our pages using the Moz Page Grader for select keywords. We've structured our site architecture appropriately for the most important pages. Additional info: We received a WMT penalty last year in march - unnatural links. We engaged an SEO agency who simply went rogue, automated link building with spun articles. We fired the agency, asked them to remove the links they had "created" and set about a resubmission. Between March and May when we made our resubmission traffic dropped 40%. Now we did have he resubmission accepted after several reconsiderations but whilst we've seen a slight recovery we have plateaued for almost a year. This penalty obviously straddles the time penguin arrived, so I wonder if perhaps we haven't seen a recovery as we are still affected by penguin. Despite all the great work we have done. If anyone has any advice or insight, please do get in touch.
Algorithm Updates | | RobertChapman0 -
How does my blog help in SEO
Hi I have recently put a wordpress blog on my site and have employed a few blog writers, each putting 2 or 3 posts per week up. There brief so far has been to write interesting, humorous and topical articles. Stupid as it may seem I have done this only because the general consensus seemed to be "you must have a blog for SEO" Does it help? Assuming it does: Should I post the same articles to my facebook page and or anywhere else? Should the articles have anchor text linking back to my site? What should I do to make it work well? Thanks in advance Andy
Algorithm Updates | | First-VehicleLeasing1 -
How can Chrome rankings vary wildly from Mozilla and IE8?
Our website (www.theideapeople.com) has generally ranked on page 4 on Google search for the phrase "charlotte web design" for the past year...regardless of which browser I used. Oh, and my browser location is always set to Charlotte, NC. I also have the Moz toolbar installed and logged-in. Today, I Googled the same keyword using Chrome and we were on page 1 in 4th position! The other browsers still showed us on page 4, as usual. Has anybody else noticed this wild variance with Chrome? Is Chrome learning that I work at The Idea People?!!
Algorithm Updates | | theideapeople0