Need only tens of pages to be indexed out of hundreds: Robots.txt is Okay for Google to proceed with?
-
Hi all,
We 2 sub domains with hundreds of pages where we need only 50 pages to get indexed which are important. Unfortunately the CMS of these sub domains is very old and not supporting "noindex" tag to be deployed on page level. So we are planning to block the entire sites from robots.txt and allow the 50 pages needed. But we are not sure if this is the right approach as Google been suggesting to depend mostly on "noindex" than robots.txt. Please suggest whether we can proceed with robots.txt file.
Thanks
-
Hi vtmoz,
Given the limitations you are telling us, I'd give noindex in robots.txt a try.
I've run some experiments and found that noindex rule in Robots.txt works. It definitely won´t remove from index that pages, but it will stop showing them for search results.I'd suggest you to try using that rule with care.
Also, run some experiments on your own. My first test would be only adding one or two pages, the one that causes more trouble being indexed (maybe due to undesired traffic or due to ranking on undesired search terms).Hope it helps.
Best luck!
GR
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why different pages rank in different countries?
Hi all, I have been investigating on why our log-in page is ranking for primary keyword, but not our homepage. I can see now homepage is ranking from our second important country. I wonder why and what causes to rank different pages in different countries for same keyword. Again the statistics does not vary much between these countries. Thnaks
Algorithm Updates | | vtmoz0 -
Product Listing Pages
Hi I had a question regarding product pages and the best way to display the page for SEO. For example, is it best to have a page for - Blue Euro Containers including a table of the capacity options you can buy.. Or, have each product split out so it has it's own product page - 60L Blue Euro Container, etc etc I know a lot of the information will be fairly similar, with the capacity being the one major difference - is this a bad thing? Some of our product tables are too big and the idea was to split them out. Thanks!
Algorithm Updates | | BeckyKey0 -
Google Search Subsections
Hi! I want to know how can I put the URL from a page like that: http://i.imgur.com/qK1NLjq.png?1 I mean: "www.calafate.com › El Chaltén" Is it possible? Thanks!!!
Algorithm Updates | | Seomediabros0 -
Site not indexed on Google UK after 4 days?
Hello!
Algorithm Updates | | digitalsoda
Wonder if anyone can help with this one? I have an ecommerce site www.doggydazzles.co.uk which went live on Friday and was submitted to Google via webmaster tools on saturday morning, but I can't find any trace of it in a google search?
I'm a bit stuck with this as its never happened to any of my other sites.
Can anyone help please or make suggestions as to what I can do to get ranked quicker? Thanks0 -
What happened on September 17 on Google?
According to mozcast: http://mozcast.com/ and to my own stats, Google had a pretty strong algorithm update on September 17. Personally I have experienced a drop of about 10% of traffic coming from Google on most of my main e-commerce site virtualsheetmusic.com. Anyone know more about that update? Any ideas about what changed? Thank you in advance for any thoughts! Best, Fab.
Algorithm Updates | | fablau1 -
Canonicalization on more than one page?
is it proper to "canocalize" more than one page in a site? Or should it only be on the home page? eg: http://www.sundayschoolnetwork.com">
Algorithm Updates | | sakeith0 -
Server Down for Few Hours went from Page 1 to Page 6?
We were on Page 1 - our server went down for about 4-6 hours and then we dropped to page 6. Would the server being down for this amount of time affect our position? Any advice would be much appreciated.
Algorithm Updates | | webdesigncwd0 -
Panda Update: Need your expertise...
Hi all, After Panda update our website lost about 45% of it's traffic from Google. It wasn't an instant drop mostly it happened gradually over the last 5 months. Our keywords (all of them except the domain name) started to lose positions from top #10 to now 40+ and all recovery attempts we have done so far didn't really help. At this moment it would be great to get some advice from the top experts like you here. What we have done so far is that We have gone through the all pages and removed the duplicate / redundant ones. We have refresh the content on the main pages and also all pages now have an canonical tags. Our website is www.PrintCountry.com. Thank you very much in advance for your time.
Algorithm Updates | | gbssinc0