Is it hurting my seo ranking if robots.txt is forbidden?
-
robots.txt is forbidden - I have read up on what the robots.txt file does and how to configure it but what about if it is not able to be accessed at all?
-
Yes, excluding certain pages can be a benefit to your rankings: if the excluded pages could be considered duplicate content with your marketing pages or with it each other.
This is usually the case for blogs (think wordpress categories) or webshops (pagination, as well as single product pages reachable by different paths (and thus having different urls). As Ryan pointed out: controll that on the page level via noindex,follow to allow PR to flow. Use noindex,nofollow for "internal" pages you dont want to see crawled.
I am not sure, but having 9950 pages indexed, but considered duplicate content might hurt rankings for other pages on that domain. Google might consider the Domain spammy.
If you need a specific hint for your domain, send me a PM and I have a look if time permits.
-
In general, I do not use robots.txt. It is a better practice to use "noindex" for the pages you do not wish to have indexed.
If I had a 10k page site with 50 marketing pages, I would either want to index the entire site, or question why the other 99% of the site exists if it does not help market the products. There are numerous challenges your scenario prevents. If you block 99% of your site with robots.txt or the noindex meta tag, you are severely disrupting the flow of PR throughout your site. Also you are either blocking content which should be indexed, or you are wasting time and resources creating junk pages on your site.
If the content truly should not be indexed, it likely should be moved to another site. I would need a lot more details about the site, it's purpose and the pages involved. Whatever the proper solution, it is not likely going to be using robots.txt to block 99% of the site.
-
So in regards to increasing ranking, is there a benefit of using the robots.txt file to only index certain "marketing" page and exclude other content that may dilute your site. For example, lets say I have 10,000 pages but only about 50 or so are my marketing page. Would using robots.txt to only crawl my main marketing pages help place emphasis on that content?
-
Sebes is correct. To add a bit more, it is not necessary to provide a robots.txt file. Actually, it is preferable in most cases not to use the file but it is necessary if you do not have direct control over the code used in every page of your site. For example, if you have a CMS or Ecommerce based site you may not have likely do not have control over many pages on your site which are automatically generated through the software. In these cases the only way you can control how crawlers will treat your site's pages is either to pay for custom modifications to your site's code or to use a robots.txt file.
-
If the robots.txt can not be read by google or bing they assume that they can crawl as much as they want to. Check out the google webmaster tool to see whether google can "see" and access your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Huge difference between GSC ranking and browser ranking for certain keywords: How to proceed?
Hi, There is a huge ranking difference between the GSC and browser for our primary keyword. As per GSC, our ranking is around 15 and when checking on the multiple different incognito browsers it's around 50. How to handle this? Which is the accurate one? Product expert from Google forums claim that what I see on browsers are the personalized results; but I tried on different browsers with different connections. Thanks
Algorithm Updates | | vtmoz0 -
Is Shopify good for SEO?
Is Shopify good for SEO? Such as indexing or ranking? Some factors: - URL is too big Some default URL structure cannot change - /product/ , /collections/, /pages/ which makes the whole url so big Need expert's suggestions.
Algorithm Updates | | BBT-Digital1 -
Prioritising SEO Tasks for Biggest Impact
Hi I want to find out, what people feel the top 3 focuses should be for an SEO (I know there are hundreds)...Content/Backlinks/Social/Technical I'm trying to better prioritise my work - I work on a large ecommerce site - with just me as the SEO & a development team in France So most of the technical stuff is controlled there. I focus on audits, KWD research & briefing content how to better optimise products. A lot of time is taken up by that as the site is so big & I'm concerned I am not putting enough effort into other areas. What should these areas be?! Backlinks/Content/Blogs - I can't do everything but would like to prioritise tasks which will have the
Algorithm Updates | | BeckyKey
biggest impact Can anyone help0 -
Domain location is a ranking factor? Back links & website?
If a website trying to rank in US and it has received many back-links from domains hosting from other countries; how it will impact website ranking? Can a website hosted in country will rank well in other country? How much the hosted location matters? Like....domain hosted in Germany but trying to rank in US?
Algorithm Updates | | vtmoz0 -
Website Rankings Dropped April 12
A client website dropped drastically on April 12. Outside of some branded keywords, search results dropped off of the first page and are buried on page 3+ at best. Nothing has changed on the site, and there were no problems with the link profile. GWT has no manual actions. Kind of at a loss. Does anyone know if there was an algorithm update or anything external that may be causing some problems here? Site is www.averybiomedical.com if you want to take a look, but I'm just curious if there was anything I should be aware of. Thanks for the help!
Algorithm Updates | | AdamWormann0 -
Is Moz Domain Authority still relvant when it comes to Google ranking?
My understanding of Moz DA is that it is predominantly based on external links. Since Penguin I am noticing more and more websites ranking high in Google with a "low" number of links and certainly a low DA but quality and relevancy of content and also of offering. I understand that there was always more to ranking than DA but is it anymore even relevant to how a site will rank in Google?
Algorithm Updates | | halloranc0 -
Added the Review rich snippet and rankings have dropped?
Hi Mozzers, I am really surprised what happened here. I conducted a small campaign by adding reviews rich snippet to 10 different microsites(covering different locations). Today I was checking if the snippets were showing up correctly in the SERPs and noticed that most of these locations with review snippet dropped in the rankings? How weird is that? Can anyone explain what happened here? Has anyone experienced this? Thank you!
Algorithm Updates | | Ideas-Money-Art0 -
Need some Real Insight into our SEO Issue and Content Generation
We have our site www.practo.com We have our blog as blog.practo.com We plan to have our main site in a months time from now as www.ray.practo.com The Issues - I will then need to direct all my existing traffic from www.practo.com to www.ray.practo.com Keeping in mind SEO and also since I will be generating new content via our Wordpress instance what are the best ways to do this so that google does not have difficulty in find out content 1. Would it be good if I put the Wordpress instance as ray.practo.com/ blog(wordpress instance comes in here in the directory) / article-url 2.Would it be better with www.practo.com / ray / blog/article-url I am using wordpress to roll out all our new SEO based content on various keywords and topics for which we want traffice - primary reasons are since we needed a content generation cms platform so that we dont have to deal with html pages and every time publish those content pages via a developer. Is the above - what soever I am planning to do in the correct manner keeping SEO in mind. Any suggestions are welcome. I seriously need to know writing seo based content on wordpress instance and have them in the urls is that a good idea? Or is only html a good idea. But we need some cms to be there so that content writers can write content independently. Please guide accordingly. Thanks
Algorithm Updates | | shanky10