Is it hurting my seo ranking if robots.txt is forbidden?
-
robots.txt is forbidden - I have read up on what the robots.txt file does and how to configure it but what about if it is not able to be accessed at all?
-
Yes, excluding certain pages can be a benefit to your rankings: if the excluded pages could be considered duplicate content with your marketing pages or with it each other.
This is usually the case for blogs (think wordpress categories) or webshops (pagination, as well as single product pages reachable by different paths (and thus having different urls). As Ryan pointed out: controll that on the page level via noindex,follow to allow PR to flow. Use noindex,nofollow for "internal" pages you dont want to see crawled.
I am not sure, but having 9950 pages indexed, but considered duplicate content might hurt rankings for other pages on that domain. Google might consider the Domain spammy.
If you need a specific hint for your domain, send me a PM and I have a look if time permits.
-
In general, I do not use robots.txt. It is a better practice to use "noindex" for the pages you do not wish to have indexed.
If I had a 10k page site with 50 marketing pages, I would either want to index the entire site, or question why the other 99% of the site exists if it does not help market the products. There are numerous challenges your scenario prevents. If you block 99% of your site with robots.txt or the noindex meta tag, you are severely disrupting the flow of PR throughout your site. Also you are either blocking content which should be indexed, or you are wasting time and resources creating junk pages on your site.
If the content truly should not be indexed, it likely should be moved to another site. I would need a lot more details about the site, it's purpose and the pages involved. Whatever the proper solution, it is not likely going to be using robots.txt to block 99% of the site.
-
So in regards to increasing ranking, is there a benefit of using the robots.txt file to only index certain "marketing" page and exclude other content that may dilute your site. For example, lets say I have 10,000 pages but only about 50 or so are my marketing page. Would using robots.txt to only crawl my main marketing pages help place emphasis on that content?
-
Sebes is correct. To add a bit more, it is not necessary to provide a robots.txt file. Actually, it is preferable in most cases not to use the file but it is necessary if you do not have direct control over the code used in every page of your site. For example, if you have a CMS or Ecommerce based site you may not have likely do not have control over many pages on your site which are automatically generated through the software. In these cases the only way you can control how crawlers will treat your site's pages is either to pay for custom modifications to your site's code or to use a robots.txt file.
-
If the robots.txt can not be read by google or bing they assume that they can crawl as much as they want to. Check out the google webmaster tool to see whether google can "see" and access your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website have Caching/Indexing / Ranking Issue
Hi, My Website (https://www.v3cars.com) is not cached or indexed on regular basic from last 15 days. before this it was cached or indexed on regular basic. We are uploading fresh content on daily basic. Currently my new content is not ranked anywhere in Google even after cached or indexed. Please help and suggest. Sandeep - Love to Cars
Algorithm Updates | | onlinesandeep0 -
Meta robots at every page rather than using robots.txt for blocking crawlers? How they'll get indexed if we block crawlers?
Hi all, The suggestion to use meta robots tag rather than robots.txt file is to make sure the pages do not get indexed if their hyperlinks are available anywhere on the internet. I don't understand how the pages will be indexed if the entire site is blocked? Even though there are page links are available, will Google really index those pages? One of our site got blocked from robots file but internal links are available on internet for years which are not been indexed. So technically robots.txt file is quite enough right? Please clarify and guide me if I'm wrong. Thanks
Algorithm Updates | | vtmoz0 -
SEO Audit after Penguin 2.1 what are you guys seeing? this is my thougts
We have looked at around 2000 sites since Penguin 2.1 launched a few weeks back. These include our customers and their own competitors site. We are going through all the data which is obviously going to take some time. Hopefully we will publish a report on our findings as we are happy to share. What I currently see in my early analysis is Roughly 70% of sites tested have 0% exact match Anchor Text for their money keywords. The other 30% have less than 5% exact match Anchor Text. The quality of the links is often still poor to the sites ranking on page 1. The content surrounding the links is only about 10-15% of the time related to the money keywords. The loading time of the sites ranking seems to not matter, we encountered a lot of slow sites. Design and usability of the site was not important. We are not seeing much impact via Social media, a lot of these sites are small business Less than 10% of sites on page 1 had a Google+ account More than 40% of page 1 sites had Facebook profiles. More than 80% of the sites ranking on page 1 had less than 100 links to the landing page that ranked What are your opinions of helping to recover if hit by the above??? Q) If you have too high an anchor text percentage and have been hit or may get hit in the future would you. a) create some more high quality links with more varied anchor text, ie Click here, brand name etc b) not create any more links just remove the links you have to dilute the anchor text c) change the anchor text on links you are able to These figures are a work in progress so data will change just wanting to share our early findings and try to get a good conversation going. What are you guys seeing?
Algorithm Updates | | tempowebdesign0 -
With regards to SEO is it good or bad to remove all the old events from our website?
Our website sells tickets for various events across the UK, we do have a LOT of old event pages on our website which simply say SOLD OUT. What is the best practice? Should these event pages be removed and a 301 redirect added to redirect to the home page? Or should these pages remain in tact with simply SOLD OUT on the page?
Algorithm Updates | | Alexogilvie0 -
Is it still possible for small businesses to rank well in google
Hi I've been playing around with ecommerce sites for a few years now and although I am no expert I'm not a complete novice. We used to do quite well in google but recent changes have halved our number of hits. I have noticed that over the last year google has given priority to large brand names as opposed to relevancy. For example, if you search for the term 'bridal jewellery' (google UK) you will see that apart from one or two the majority of placements are taken by big compnies who offer very little bridal jewellery. One or two pages at most. My question is, is it still possible to rank well against these brand names or has google made it impossible for small companies. PS we only practice ethical seo as suggested by seomoz. Any help or advice is greatly appreciated. Thanks www.kerryblu.co.uk
Algorithm Updates | | Dill0 -
Video SEO: Youtube, Vimeo PRO, Wistia, Longtail BOTR Experience and questions
Obviously Video SEO is changing, Google is figuring out how to do it themselves. We are left wondering… Below we have tried to explain what we have learned and how the different sites work and their characteristics (links to graphics provided) Our problem is: We are not getting congruent Google site:apalytics.tv Video filter results. We are wondering how duplicate content may be affecting our results… and if so, why will Youtube not be duplicate and prevent your own site SEO efforts from working. Is Youtube special? Does that include Vimeo too? We see our own duplicate videos on multiple sites in Google results, so it seems it is not duplicate related…? We’d appreciate your experience or add to our questions and work as a community to get this figured out more definitively. Thanks! We’ve tried four video hosting solutions at quite a cost monetarily and in time. 1.) Youtube, which gets all the SEO Juice and gets our clients on to other subjects or potentially competitive content. Iframes just don’t get the results we are looking for. 2.) See Vimeo Image: Vimeo PRO, a $200 year plus solution that allows us to do many video carousels on our own domains hosted on Vimeo, but are very limited in HTML as only CSS content changes are allowed. While we were using Vimeo we allowed the Vimeo.com community to SEO our content directly and they come up often in search results. Due to duplicate content concerns we have disallowed Vimeo.com from using our content and SEOing our content to their domain. However, we have many “portfolios” (micro limited carousal sites on our domains) that continue to carry the content. The Vimeo hosted micro site shows only three videos on Google: site:apalytics.tv During our testing we are concerned that duplicate content is causing issues too, so we are getting ready to shut off the many microsite domains hosted at Vimeo. (Vimeo has an old embed code that allows a NON-iframe embed – but has discontinued it recently) That makes it difficult if not impossible to retain SEO juice for anything other than their simple micro sites that are very limited! 3.) See Wistia Image: Wistia, a $2000 year plus solution that only provides private video site hosting embedding various types of video content on one’s site/s. Wistia has a free account now for three videos and limited plays – it’s a nice interface for SEO but is still different than BOTR. We opted for BOTR because of many other advertising related options, but are again trying Wistia with the free version to see if we can figure out why our BOTR videos are not showing up as hoped. We know that Google does not promise to index and feature every video on a sitemap, but why some are there and others are not and when remains a mystery that we are hoping to get some answers about. 4.) See Longtail Image: Longtail, Bits On The Run, (JW Player author) a $1,000 year plus like Wistia provides private hosting, but it allows a one button YouTube upload for the same SEO meta data and content – isn’t that duplicate content? BOTR creates and submits video sitemaps for your content, but it has not been working for us and it has been impossible to get a definitive answer as I think they too are learning or are not wanting the expose their proprietary methods (which are not yet working for us!) 2O9w0.png 0eiPv.png O9bXV.png
Algorithm Updates | | Mark_Jay_Apsey_Jr.0 -
High ranking for high volume keyword, but low traffic
We are ranked, according to Moz (and we've tested to back it up) #3 on Google UK for the keyword "Hire a Jet". According to Google, this keyword gets 22,500 local searches per month. Yet we get about 5 hits a month for that keyword. Any ideas why this is so low? It just doesn't add up or make sense whatsoever.
Algorithm Updates | | JetBookMike0