Is it hurting my seo ranking if robots.txt is forbidden?
-
robots.txt is forbidden - I have read up on what the robots.txt file does and how to configure it but what about if it is not able to be accessed at all?
-
Yes, excluding certain pages can be a benefit to your rankings: if the excluded pages could be considered duplicate content with your marketing pages or with it each other.
This is usually the case for blogs (think wordpress categories) or webshops (pagination, as well as single product pages reachable by different paths (and thus having different urls). As Ryan pointed out: controll that on the page level via noindex,follow to allow PR to flow. Use noindex,nofollow for "internal" pages you dont want to see crawled.
I am not sure, but having 9950 pages indexed, but considered duplicate content might hurt rankings for other pages on that domain. Google might consider the Domain spammy.
If you need a specific hint for your domain, send me a PM and I have a look if time permits.
-
In general, I do not use robots.txt. It is a better practice to use "noindex" for the pages you do not wish to have indexed.
If I had a 10k page site with 50 marketing pages, I would either want to index the entire site, or question why the other 99% of the site exists if it does not help market the products. There are numerous challenges your scenario prevents. If you block 99% of your site with robots.txt or the noindex meta tag, you are severely disrupting the flow of PR throughout your site. Also you are either blocking content which should be indexed, or you are wasting time and resources creating junk pages on your site.
If the content truly should not be indexed, it likely should be moved to another site. I would need a lot more details about the site, it's purpose and the pages involved. Whatever the proper solution, it is not likely going to be using robots.txt to block 99% of the site.
-
So in regards to increasing ranking, is there a benefit of using the robots.txt file to only index certain "marketing" page and exclude other content that may dilute your site. For example, lets say I have 10,000 pages but only about 50 or so are my marketing page. Would using robots.txt to only crawl my main marketing pages help place emphasis on that content?
-
Sebes is correct. To add a bit more, it is not necessary to provide a robots.txt file. Actually, it is preferable in most cases not to use the file but it is necessary if you do not have direct control over the code used in every page of your site. For example, if you have a CMS or Ecommerce based site you may not have likely do not have control over many pages on your site which are automatically generated through the software. In these cases the only way you can control how crawlers will treat your site's pages is either to pay for custom modifications to your site's code or to use a robots.txt file.
-
If the robots.txt can not be read by google or bing they assume that they can crawl as much as they want to. Check out the google webmaster tool to see whether google can "see" and access your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can you rank without spending lots of money?
Hello Everyone, This is a general question, and its one I have been thinking about recently because I am working on promoting couple of websites. I want to know if it is still possible to make websites rank consistently without spending lots of money. I am self employed, and 5 or so years ago, I remember I did all my own link building/content for about 6 or so websites that I owned, and I managed to make most of them rank consistently. I am not in the SEO business, and I have not touched any SEO type of work for about 5 years now. And I always did it for my own websites/business. I know this is a fast moving industry, and my general knowledge may be a bit out of date. I kind of get the feeling that the days of when small business owners could make a website rank on a shoe string budget and make a bit of money with an online business may have died or is dying. I am a realist and I know that only a very tiny percentage of websites make really good quality/fresh content that everyone wants to reads. I know a common advise is that you should create a site with such amazing content that everyone talks about you and mentions/links to you without you even to need to do any link building. But in my opinion (I could be wrong), but I feel that this probably happens to less than 0.01 percent of websites. And I also know even third rate websites with blogs or content sites charge to post an article with a link. So this makes me think that nowadays you need a good budget and plenty of time to make a website rank. Am I wrong? In today's internet, do you need to spend money to rank? I genuinely want to know peoples experience and or opinion on this subject. Thanks.
Algorithm Updates | | Ryan.Shahed0 -
Does using parent pages in WordPress help with SEO and/or indexing for SERPs?
I have a law office and we handle four different practice areas. I used to have multiple websites (one for each practice area) with keywords in the actual domain name, but based on the recommendation of SEO "experts" a few years ago, I consolidated all the webpages into one single webpage (based on the rumors at the time that Google was going to be focusing on authorship and branding in the future, rather than keywords in URLs or titles). Needless to say, Google authorship was dropped a year or two later and "branding" never took off. Overall, having one webpage is convenient and generally makes SEO easier, but there's been a huge drawback: When my page comes up in SERPs after searching for "attorney" or "lawyer" combined with a specific practice area, the practice area landing pages don't typically come up in the SERPs, only the front page comes up. It's as if Google recognizes that I have some decent content, and Google knows that I specialize in multiple practice areas, but it directs everyone to the front page only. Prospective clients don't like this and it causes my bounce rate to be high. They like to land on a page focusing on the practice area they searched for. Two questions: (1) Would using parent pages (e.g. http://lawfirm.com/divorce/anytown-usa-attorney-lawyer/ vs. http://lawfirm.com/anytown-usa-divorce-attorney-lawyer/) be better for SEO? The research I've done up to this point appears to indicate "no." It doesn't make much difference as long as the keywords are in the domain name and/or URL. But I'd be interested to hear contrary opinions. (2) Would using parent pages (e.g. http://lawfirm.com/divorce/anytown-usa-attorney-lawyer/ vs. http://lawfirm.com/anytown-usa-divorce-attorney-lawyer/) be better for indexing in Google SERPs? For example, would it make it more likely that someone searching for "anytown usa divorce attorney" would actually end up in the divorce section of the website rather than the front page?
Algorithm Updates | | micromano0 -
Ranking Well in Google But Not Bing - Why?
Hello Moz Community, I'm ranking well in Google (#2-#6 for various keywords) but on the second page of Bing. Are there certain differences that I should be aware of? Thanks, Cole
Algorithm Updates | | ColeLusby0 -
Website dropping from Page 1 pos 5 to no ranking and then back again?
Hi all, We have a very odd occurrence with a client of ours. It should be noted that they had a penalty recently removed about 2 months ago after much work from our company. Recently they started appearing back on Page 1 Google for a semi competitive keyword term. We were very happy with this and so was the client. The the ranking improved with our work to position 5, which was excellent. Unfortunately what has been happening is they have been dropping out of the rankings completely for this semi competitive keyword for a few days and then reappearing in the same position. The client is checking daily and has noticed. I thought this is just a 'hangover' from the Google penalty and perhaps a one off occurrence, but it has happened about 3 or 4 times now and seems to be happening every couple of weeks. Can anyone shed some light on this behavior? I have checked Webmasters Tools and everything is fine. Thanks Jon
Algorithm Updates | | Jon_bangonline0 -
Absolute Internal Links and their Affect on Rankings for Local Businesses
For many localized search terms (City + Profession), I'm noticing that OpenSiteExplorer is reporting high numbers of backlinks for anchortext from Absolute internal links from that same domain. Now, I'm familiar with the difference in relative and absolute internal links but am wondering if this type of linking may be carrying a lot of weight for rankings. It may be a correlation over causation situation if it's more that those companies that are including absolute internal links for terminology are just doing a better job (generally) with internal linking... but I feel like this may be something worth digging into. Everything I have read says that search engines view absolute and relative internal links essentially equally but does anybody have their own insight on the effectiveness for these two types of internal links in regards to small local businesses who otherwise are getting basically no links at all?
Algorithm Updates | | Kirch0 -
Sudden Page Rank Drop for Weight Loss Camp
My site BalanceME.com has been performing really well and continually climbs more and more but in the last 7 days we have had a drop this last week. We went from #2 for weight loss camp to #6, and even further on other terms. It is mid May and I cannot find any other explanation for the sudden drop. I am hoping someone could give me some sort if idea what could possibly be the answer. Recently we have had two press releases go out, target for keywords, and also installed images that navigate to new pages on the site. There have been few additions to the site, and nothing can explain the drop. Other companies in our group have not had issues with recent drops, and we use the same practices with the other organizations. Thank you for your time
Algorithm Updates | | FVdBeuken0 -
What determines rankings in a site: search?
When I perform a "site:" search on my domains (without specifying a keyword) the top ranked results seem to be a mixture of sensible top-level index pages plus some very random articles. Is there any significance to what Google ranks highly in a site: search? There is some really unrepresentative content returned on page 1, including articles that get virtually no traffic. Is this seriously what Google considers our best or most typical content?
Algorithm Updates | | Dennis-529610