Is it hurting my seo ranking if robots.txt is forbidden?
-
robots.txt is forbidden - I have read up on what the robots.txt file does and how to configure it but what about if it is not able to be accessed at all?
-
Yes, excluding certain pages can be a benefit to your rankings: if the excluded pages could be considered duplicate content with your marketing pages or with it each other.
This is usually the case for blogs (think wordpress categories) or webshops (pagination, as well as single product pages reachable by different paths (and thus having different urls). As Ryan pointed out: controll that on the page level via noindex,follow to allow PR to flow. Use noindex,nofollow for "internal" pages you dont want to see crawled.
I am not sure, but having 9950 pages indexed, but considered duplicate content might hurt rankings for other pages on that domain. Google might consider the Domain spammy.
If you need a specific hint for your domain, send me a PM and I have a look if time permits.
-
In general, I do not use robots.txt. It is a better practice to use "noindex" for the pages you do not wish to have indexed.
If I had a 10k page site with 50 marketing pages, I would either want to index the entire site, or question why the other 99% of the site exists if it does not help market the products. There are numerous challenges your scenario prevents. If you block 99% of your site with robots.txt or the noindex meta tag, you are severely disrupting the flow of PR throughout your site. Also you are either blocking content which should be indexed, or you are wasting time and resources creating junk pages on your site.
If the content truly should not be indexed, it likely should be moved to another site. I would need a lot more details about the site, it's purpose and the pages involved. Whatever the proper solution, it is not likely going to be using robots.txt to block 99% of the site.
-
So in regards to increasing ranking, is there a benefit of using the robots.txt file to only index certain "marketing" page and exclude other content that may dilute your site. For example, lets say I have 10,000 pages but only about 50 or so are my marketing page. Would using robots.txt to only crawl my main marketing pages help place emphasis on that content?
-
Sebes is correct. To add a bit more, it is not necessary to provide a robots.txt file. Actually, it is preferable in most cases not to use the file but it is necessary if you do not have direct control over the code used in every page of your site. For example, if you have a CMS or Ecommerce based site you may not have likely do not have control over many pages on your site which are automatically generated through the software. In these cases the only way you can control how crawlers will treat your site's pages is either to pay for custom modifications to your site's code or to use a robots.txt file.
-
If the robots.txt can not be read by google or bing they assume that they can crawl as much as they want to. Check out the google webmaster tool to see whether google can "see" and access your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Which is the best way to rank a site?
Hi, I have been working on SEO for a long time, recently I started a new site where I was aiming to rank different niches but I am stuck. First I covered some keywords related to sports then I shifted the niche to hunting. My idea was to cover a niche fully then move on to the 2nd so the authority of the site can also help rank the 2nd niche but the problem is I am unable to rank my site. Should I be considering only a very specific niche site or should I continue doing all the stuff on the same site. Please checkout my site ReviewsCase.com and let me know. And if has also done the same please let me know.
Algorithm Updates | | seoasikhan20 -
New Google Update - weird ranking
Hi I wanted to get your thoughts on this keyword ranking. This page - https://www.key.co.uk/en/key/albatross-heavy-duty-office-chairs-24-stone is now ranking for heavy duty office chair 30 stone We don't mention 30 in the content anywhere, apart from the USPs at the top of the page - could this be it?! I don't know how to change this, or I guess Google is still figuring things out and maybe this will drop off? Love to get some thoughts on this! Becky
Algorithm Updates | | BeckyKey0 -
Do keyword target landing pages increase rankings?
Let's say we create landing pages for targeted keywords in our niche. So like we have landing pages optimised for 80% of the top keywords with decent search volume. If these pages started ranking at first page or around; will this scenario improves the ranking of website? Right now, only few of our top pages are ranking good. Planning to create more of such.
Algorithm Updates | | vtmoz0 -
Could Retail Price Be A Google Ranking Factor???
I have not done any detailed studies on this but it seems that Google might be using low retail prices for specific items as a ranking factor in their organic SERPs. Does anyone else suspect this? Just askin' to hear your thoughts. Thanks!
Algorithm Updates | | EGOL0 -
Merging Multiple Domains into a Single Domain and Its Effect on Ranking
My client had multiple top-level-domains. Each one represented an insurance program within a specific vertical. For all the sites at these alternate domains, there was a 30/70 mix of duplicate vs. original content. Some of the alternate domains ranked very well for their target keyphrase groups, where others were absent in results pages. We advised the client to merge multiple domains into their existing main domain, for usability and SEO reasons. We recently ran the merger. Here was our process: On the main domain, transfer the content such that it matches 1-for-1 content on the various alternate domains Setup Google Webmaster tools on the main domain Push the new content on the main domain live and submit a corresponding sitemap to Google Establish 301 redirects on the alternate domains, such that each alternate domain URL points to its respective page on the main domain We did this 12 days ago, and pages (previously on the alternate domains) that had ranked well on Google have now plummeted or are entirely non-existent. Did we do the right thing by merging multiple top-level domains into a single domain? Is this initial dip in rankings normal? How soon should we expect to see it return to its normal rankings?
Algorithm Updates | | PinckneyHugoGroup0 -
Help with local Seo?
Hi, I am really struggling with current predicament i find myself in. I am a small to medium sized business based in Newcastle in the UK and am trying to rank well locally for the keywords that i feel my customers will be searching for locally. I have got to the stage where i am on page 1 of google uk or nearly there but cannot compete against the national companies who have the search terms then just add pages for virtually every city in the country. For example my main product is "Artificial Grass" and my city/town is Newcastle This is where my office is and where my customers are. This is also where my google places page states. Now theres a company that sells Artificial Grass called www.asgoodasgrass.co.uk that are based no where near but use the power of there site to come up in every local search by adding a page "Artificial Grass Newcastle" as well as hundreds of others. They rank 3rd and im 8th. There actual Newcastle page is poor, where as i put everything into my page including pics, video etc. Still no joy. I feel i am always going to rank behind these big boys even though i am the actual local company and have no intention of working others area that are not local to me. By the time i rank behind the above type companies and the likes of yell.com i feel i am never going to be seen and fall back on expensive adwords to help me along. I am a complete newbie at this and would love any help or tips you could give to give me a fighting chance in my area. My site is www.totaldrivewaysne.co.uk incase you want to look as you will have gathered my other primary product is driveways for which also i feel like i have a million competitors! many many thanks for any responses John
Algorithm Updates | | totaldriveways0 -
Do you believe CTR affects rankings
As you know, Google tracks CTR on keywords for your website visible through google Webmaster Tools. How much if at all do you believe this affects your rankings on the tracked keywords?
Algorithm Updates | | TheGrid0 -
Can I check the rank of a keyword over time ?
Is there any tools available that allows users to track ranking of a keyword in SERP over time ? I know the question can be a bit confusing so here is an example that I hope makes it a bit easier to understand EXAMPLE : I am doing keyword research for say "iphones games" and I find out the current sites that rank for the term but If I want to see who ranked for the term 6 months ago or 1 year ago, is it possible ? Also can I get data of the SERP ranking history for "example.com" for the term "iphones games" ? eg : in jan 2011 rank 10 feb 2011 rank 7 ... sep 2011 rank 5
Algorithm Updates | | avant_seomoz0