Is it hurting my seo ranking if robots.txt is forbidden?
-
robots.txt is forbidden - I have read up on what the robots.txt file does and how to configure it but what about if it is not able to be accessed at all?
-
Yes, excluding certain pages can be a benefit to your rankings: if the excluded pages could be considered duplicate content with your marketing pages or with it each other.
This is usually the case for blogs (think wordpress categories) or webshops (pagination, as well as single product pages reachable by different paths (and thus having different urls). As Ryan pointed out: controll that on the page level via noindex,follow to allow PR to flow. Use noindex,nofollow for "internal" pages you dont want to see crawled.
I am not sure, but having 9950 pages indexed, but considered duplicate content might hurt rankings for other pages on that domain. Google might consider the Domain spammy.
If you need a specific hint for your domain, send me a PM and I have a look if time permits.
-
In general, I do not use robots.txt. It is a better practice to use "noindex" for the pages you do not wish to have indexed.
If I had a 10k page site with 50 marketing pages, I would either want to index the entire site, or question why the other 99% of the site exists if it does not help market the products. There are numerous challenges your scenario prevents. If you block 99% of your site with robots.txt or the noindex meta tag, you are severely disrupting the flow of PR throughout your site. Also you are either blocking content which should be indexed, or you are wasting time and resources creating junk pages on your site.
If the content truly should not be indexed, it likely should be moved to another site. I would need a lot more details about the site, it's purpose and the pages involved. Whatever the proper solution, it is not likely going to be using robots.txt to block 99% of the site.
-
So in regards to increasing ranking, is there a benefit of using the robots.txt file to only index certain "marketing" page and exclude other content that may dilute your site. For example, lets say I have 10,000 pages but only about 50 or so are my marketing page. Would using robots.txt to only crawl my main marketing pages help place emphasis on that content?
-
Sebes is correct. To add a bit more, it is not necessary to provide a robots.txt file. Actually, it is preferable in most cases not to use the file but it is necessary if you do not have direct control over the code used in every page of your site. For example, if you have a CMS or Ecommerce based site you may not have likely do not have control over many pages on your site which are automatically generated through the software. In these cases the only way you can control how crawlers will treat your site's pages is either to pay for custom modifications to your site's code or to use a robots.txt file.
-
If the robots.txt can not be read by google or bing they assume that they can crawl as much as they want to. Check out the google webmaster tool to see whether google can "see" and access your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Have you ever seen or experienced a page indexed which is actually from a website which is blocked by robots.txt?
Hi all, We use robots file and meta robots tags for blocking website or website pages to block bots from crawling. Mostly robots.txt will be used for website and expect all the pages to not getting indexed. But there is a condition here that any page from website can be indexed by Google even the site is blocked from robots.txt; because crawler may find the page link somewhere on internet as stated here at last paragraph. I wonder if this really the case where some webpages have got indexed. And even we use meta tags at page level; do we need to block from robots.txt file? Can we use both techniques at a time? Thanks
Algorithm Updates | | vtmoz0 -
SEO for International Expansion - Best Strategies
Looking for some Best Strategies in SEO for International Expansion . We were based in a single country till one year back and then decided to expand to other English speaking countries as well. Traffic from SEO for us is around 1.3 million users only from the single country and we rank in top#3 for 90% of our targeted keyword bank of 45000+ keywords. We just launched in some new countries like singapore , philippines and few other English speaking countries with really unique content and I wanted to check whether I can use my mother domain or use any other strategies for growing SEO faster in other countries. Any inputs on scaling up SEO traffic internationally will be appreciated
Algorithm Updates | | ozil1 -
Do links from unrelated sites dilute your rankings for your key phrases?
do links from unrelated sites dilute your rankings for your key phrases? i've always heard don't get links from unrelated sites but if that mattered, then how would sites with totally diverse pages such as newspaper sites, sears, and other catalogue sites rank for these diverse subjects on their site? How does Facebook rank when it gets 100,000 links a day from sites that have nothing to do with a social media site? I'd love to hear everyone's opinion on this. Also, Do links from unrelated sites give less push than related links? Take care,
Algorithm Updates | | Ron10
Ron0 -
Significant Drop in Rankings without Explanation
My company has experienced a significant drop in rankings in the last month or so. How significant? Across our top 11 keywords, we've dropped an average of 6 positions. Some have dropped less (1 or 2) and some have dropped way more (38). We work really hard to provide great content on our site and have been building out our link profile with guest blogging on relevant sites (dailyshotofcoffee.com, for example). No major changes have been made to the URL structure or content on the pages that are ranking for our key terms, so I am not sure where the drop is coming from. For example: One of our key terms is "espresso machine" and we went from #9 to #22 in the last few weeks. We have not made any changes to the main content on the page that is ranking since September. Our on-site page report for this page has us nailing all of the critical factors, most of the high importance factors (not exact keyword in page titles - we use "espresso machines" as that is one of our other terms - or "avoid keyword stuffing" - unfortunately, our products are also named with "espresso machine" and i can't very well not follow those links or remove the term from their names), all of the moderate importance factors, and most of the low importance factors. It has been reporting this way since September. Most of our key terms are this way (haven't had content changed in the last several weeks to months and have great on-page grades). We don't engage in spammy link building (though I did some work this fall on cleaning up bad links a previous SEO company had built out). I'm just really taken aback by the sudden drop in rankings across the board. Any insight or advice anyone can give me would be greatly appreciated!
Algorithm Updates | | Marketing.SCG1 -
Is anybody else seeing large scale rankings drops in Bing this week?
I track around 1000 keywords for this site, and my rankings in Bing dropped for about half of them on Wednesday. No major changes have been made to the site, rankings are maintaining or improving in Google for a majority of these same terms. The average drop seems to be around 9-12 places, which to me signals more than just standard fluctuation. Anyone else seeing anything strange with Bing this week? Or does anyone have any ideas? I looked for posts about an algorithm change but haven't found anything. Thanks.
Algorithm Updates | | BrianCC0 -
Is using WPML (WordPress Multilingual Plugin) ok for On-Page SEO?
Hi Mozzers, I'm investigating multilingual site setup and translating content for a small website for 15-20 pages and came accross WPML (WordPress Multilingual Plugin) which looks like it could help, but I am curious as to whether it has any major international SEO limitations before trialing/buying. It seems to allow the option to automatically setup language folder structures as www.domain.com/it/ or www.domain.com/es/ etc which is great and seems to offer easy way of linking out to translators (for extra fee), which could be convenient. However what about the on-page optimization - url names, title tags and other onpage elements - I wonder if anyone has any experiences with using this plugin or any alternatives for it. Hoping for your valued advice!
Algorithm Updates | | emerald0 -
What determines rankings in a site: search?
When I perform a "site:" search on my domains (without specifying a keyword) the top ranked results seem to be a mixture of sensible top-level index pages plus some very random articles. Is there any significance to what Google ranks highly in a site: search? There is some really unrepresentative content returned on page 1, including articles that get virtually no traffic. Is this seriously what Google considers our best or most typical content?
Algorithm Updates | | Dennis-529610 -
Singular vs plural SEO
Hi everyone, OK I've been looking at the Google adwords keyword tool and it's thrown some of my On-page SEO into question (everything said here are examples, I haven't used any real life terms or figures). Lets say my page is about "Green Apples", let's say the keyword tool shows that the singular version "Green Apple" gets more searches (as an example). Should I optimize for the singular or the plural? Also lets say my title tag for that page is "Green Apples | Apples Galore UK" would Google/SEOmoz count that as an optimisation for the singular "Green Apple" or do the search engines take the title literally and don't differenciate between singular and plurals? Thanks in advance everyone! Regards, Ash
Algorithm Updates | | AshSEO20112