Is it hurting my seo ranking if robots.txt is forbidden?
-
robots.txt is forbidden - I have read up on what the robots.txt file does and how to configure it but what about if it is not able to be accessed at all?
-
Yes, excluding certain pages can be a benefit to your rankings: if the excluded pages could be considered duplicate content with your marketing pages or with it each other.
This is usually the case for blogs (think wordpress categories) or webshops (pagination, as well as single product pages reachable by different paths (and thus having different urls). As Ryan pointed out: controll that on the page level via noindex,follow to allow PR to flow. Use noindex,nofollow for "internal" pages you dont want to see crawled.
I am not sure, but having 9950 pages indexed, but considered duplicate content might hurt rankings for other pages on that domain. Google might consider the Domain spammy.
If you need a specific hint for your domain, send me a PM and I have a look if time permits.
-
In general, I do not use robots.txt. It is a better practice to use "noindex" for the pages you do not wish to have indexed.
If I had a 10k page site with 50 marketing pages, I would either want to index the entire site, or question why the other 99% of the site exists if it does not help market the products. There are numerous challenges your scenario prevents. If you block 99% of your site with robots.txt or the noindex meta tag, you are severely disrupting the flow of PR throughout your site. Also you are either blocking content which should be indexed, or you are wasting time and resources creating junk pages on your site.
If the content truly should not be indexed, it likely should be moved to another site. I would need a lot more details about the site, it's purpose and the pages involved. Whatever the proper solution, it is not likely going to be using robots.txt to block 99% of the site.
-
So in regards to increasing ranking, is there a benefit of using the robots.txt file to only index certain "marketing" page and exclude other content that may dilute your site. For example, lets say I have 10,000 pages but only about 50 or so are my marketing page. Would using robots.txt to only crawl my main marketing pages help place emphasis on that content?
-
Sebes is correct. To add a bit more, it is not necessary to provide a robots.txt file. Actually, it is preferable in most cases not to use the file but it is necessary if you do not have direct control over the code used in every page of your site. For example, if you have a CMS or Ecommerce based site you may not have likely do not have control over many pages on your site which are automatically generated through the software. In these cases the only way you can control how crawlers will treat your site's pages is either to pay for custom modifications to your site's code or to use a robots.txt file.
-
If the robots.txt can not be read by google or bing they assume that they can crawl as much as they want to. Check out the google webmaster tool to see whether google can "see" and access your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does using non-https links (not pages) impact or penalise the website rankings?
Hi community, We have couple of pages where we we have given non-https (http) hyperlinks by mistake. They will redirect to http links anyway. Does using these http links on page hurt any rankings? Thansk
Algorithm Updates | | vtmoz0 -
Help - Losing Ranking on Plural Keywords in Google
Help! I'm losing keyword rankings for certain keywords - but still ranking well on others. The trend I can see is that we're losing rankings on plurals, so this page https://www.key.co.uk/en/key/platform-trolleys-trucks Used to rank well for: platform trolleys
Algorithm Updates | | BeckyKey
platform trucks
flat bed trolleys But now only ranks well for platform trolley flat bed trolleys platform truck I have seen a large number of keywords drop out - they're not hitting traffic massively yet, but conversions are dropping - help! I;m struggling to find the cause!0 -
Will more landing pages with related keywords improve ranking of main keyword?
Hi all, We are in the process of creating new landing pages and noticed something what our competitors been following with landing pages. They have created multiple landing pages targeting the related keywords of a main topic / main keyword. Like "seo defined", "seo history", "seo benefits", "seo strategies", "types of seo", etc being "SEO" as main keyword or main topic. Will this works? Thanks
Algorithm Updates | | vtmoz0 -
New feature in seo results with icon?
I have never seen it before in the search: an icon in the title. Do you guys know how to get this icon in the title? See here: http://snag.gy/e7BiI.jpg e7BiI.jpg
Algorithm Updates | | Emilija1 -
Absolute Internal Links and their Affect on Rankings for Local Businesses
For many localized search terms (City + Profession), I'm noticing that OpenSiteExplorer is reporting high numbers of backlinks for anchortext from Absolute internal links from that same domain. Now, I'm familiar with the difference in relative and absolute internal links but am wondering if this type of linking may be carrying a lot of weight for rankings. It may be a correlation over causation situation if it's more that those companies that are including absolute internal links for terminology are just doing a better job (generally) with internal linking... but I feel like this may be something worth digging into. Everything I have read says that search engines view absolute and relative internal links essentially equally but does anybody have their own insight on the effectiveness for these two types of internal links in regards to small local businesses who otherwise are getting basically no links at all?
Algorithm Updates | | Kirch0 -
Trying to recover from a ranking crash
My site was well established and I'd built up some good rankings in Google using link building and onsite optimization. Then last year it completely fell out of the rankings except for a few keywords. There were some suspicious links but nothing that seemed too toxic, no warnings from Webmaster Tools. 9 months later I still haven't recovered. The site is well optimized. Any ideas?
Algorithm Updates | | Infogenix0 -
Can visitors duration time affect Google Rankings?
Does the time a person stays on a website affect Search Rankings? If so, could the lower time from Adwords Visitors be effecting organic rankings? And the same for bounce rate. If Non-Paid Search Traffic Avg. Visit Duration time is 3:55 and Paid Search Traffic Avg. Visit Duration is 1:59 Could the low duration time be affecting our website rankings?
Algorithm Updates | | hfranz0 -
Another Domain ranking instead of my Domain
Hi My Domain name is Replicahause.net, 2 weeks ago my server had an outage for 3 days and my rankings dissappeared in google entirely, however i also noticed that when i typed in my domain name "replicahause" or "replicahause.net" , i would see abhishekyadav.com appearing on #1 in google which does a 301 into Replicahause.net I was able to convince the owner of Abhishekyadav to remove the 301 but my site Replicahause.net's Rankings still does not appear to have come back to google, is there something i'm missing here ? We were ranked #1 to #10 for at least 40 keywords, they've just seemed to dissappeard after the server downtime we had and the 301 from AbhishekYadav.com Thanks
Algorithm Updates | | jansimon0