Need only tens of pages to be indexed out of hundreds: Robots.txt is Okay for Google to proceed with?
-
Hi all,
We 2 sub domains with hundreds of pages where we need only 50 pages to get indexed which are important. Unfortunately the CMS of these sub domains is very old and not supporting "noindex" tag to be deployed on page level. So we are planning to block the entire sites from robots.txt and allow the 50 pages needed. But we are not sure if this is the right approach as Google been suggesting to depend mostly on "noindex" than robots.txt. Please suggest whether we can proceed with robots.txt file.
Thanks
-
Hi vtmoz,
Given the limitations you are telling us, I'd give noindex in robots.txt a try.
I've run some experiments and found that noindex rule in Robots.txt works. It definitely won´t remove from index that pages, but it will stop showing them for search results.I'd suggest you to try using that rule with care.
Also, run some experiments on your own. My first test would be only adding one or two pages, the one that causes more trouble being indexed (maybe due to undesired traffic or due to ranking on undesired search terms).Hope it helps.
Best luck!
GR
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Great DA but page authority not increasing!
Hey team, I hope you are doing great, I have been working effortlessly to increase the authority of my blog. I have used a number of Moz recommended methods like long-form content, posting frequency, getting references from influencers and great websites. It has all resulted in a good domain authority but no matter what I do, the page authority of my blog isn't increasing. Can you please have a look and guide: https://androidcompare.com/ Kind regards...
Algorithm Updates | | Bsmdi0 -
Key webpage fluctuating between page 2 and page 6 of Google SERP
Hi, We have found that one of our key webpages has been fluctuating between page 2 and page 6 of Google SERP for around 2 weeks. Some days it will be on page 6 in the morning and then page 2 in the afternoon. We have recently updated some copy on the page and wondered if this could be the cause. Has anyone else experienced this? If so how long was it before the page settled? https://www.mrisoftware.com/uk/products/property-management-software/ Thanks.
Algorithm Updates | | nfrank0 -
Google update January 2015
Hello, In January 2015, google changed its European Algorithm. The change decreased the ranking of some of our keywords but not all. See article for more evidence in google changing its algorithm. https://www.seroundtable.com/google-update-maybe-19760.html The biggest change was the keyword phrase ‘Wholesale Silver Jewellery’ which we ranked 1 in SERP, but now we’re nowhere to be seen. However, the change didn’t affect our keyword phrase ‘Wholesale Jewellery Silver’, ’Wholesale Silver’ and ‘Wholesale Jewellery. We’ve been through our data and see that all of our ’Silver Jewellery’ keyword phrases are no longer showing in the SERP. Further research has shown that our competitors were also dropped down the rankings for the same keyword phrase. Our question is: Why has this update affected certain keyword phrases, such as ‘silver jewellery’ but not ‘jewellery silver’ and how should we over come this? Additional Information
Algorithm Updates | | SilverStar1
If you type in our company name ‘Mainly Silver’ or ‘mainlysilver’ were still showing in SERP, however if you type ‘mainlysilver jewellery’ we’re no where to be found. We’ve even checked ‘site:mainlysilver.co.uk silver jewellery’ in google search and it returns with ‘no results found’. If you switch the keyword phrase, all our web pages are showing up Our website is - www.mainlysilver.co.uk0 -
Anyone Notice Google's Latest Change Seems to Favor Google Books?
I've noticed a change in the search results lately. As I search around I notice a lot of results from books.google.com Seems a little (ok a lot) self serving... JMHO
Algorithm Updates | | get4it1 -
Same Meta description is being shown on Google?
Not sure why this is happening but when you this command into Google site:"mywebsite": + "key phrase" It brings up pages from my website which have the key phrase but I have noticed that Google is using the wrong meta description for all of them even though these pages all have their own unique meta description Does anyone know why this would be happening? Thanks
Algorithm Updates | | webguru20140 -
Will Google discount new gTLDs?
I have the opportunity now to acquire a very desirable generic domain name with either the .org or .pro TLD. Obviously, the .pro version I can get for a far better price. The .org will probably end up being about 10x the price of the .pro. I feel like the .org would give me instant clout while the .pro might raise eyebrows. Also, I'm concerned that Google might also discount these new gTLDs. What do you guys think? Is the perceived authority of the old-time TLDs something worth investing in? Or will this fizzle away over time as the new gTLDs flood the market? THanks! Ira
Algorithm Updates | | iraweissman0 -
MOZ.com Page Rank of 2?
I don't recall the page rank of SEOMoz.com prior to the company's change to MOZ.com. But did notice that MOZ.com currently has a Page Rank of 2 (which I find weird since it's such a strong, content rich, highly-regarded site). I'd be interested in hearing about findings from the MOZ.com team on why the low PR and how has it affected your site since the change? (...and perhaps a look at the future through a crystal ball 🙂 I recall reading the MOZ domain changing article titled "Domain Migrations: Surviving the "Perfect Storm" of Site Changes" which had great info and addresses some reasons for PR loss in the 'Traffic and Ranking Loss' section: http://moz.com/blog/domain-migration-lessons
Algorithm Updates | | Prospector-Plastics0 -
Been Penalized, Starting from Scratch, Need Advice
Good Morning, We've been penalized for unnatural links on the site : http://goo.gl/JgK1e After attempting to contact many of the sites with links pointing at our site, and getting no response, we decided to start from scratch on a new domain : http://goo.gl/XUH3f The first thing I did once the new domain was up was remove all of the unique content (text) from the original site and place it on the new site... I am still having a difficult time getting the new site to rank in the SERPS. Can you guys please provide pointers as to what steps should be taken to get the new domain, http://goo.gl/XUH3f a high ranking in the SERPS? Thanks!!
Algorithm Updates | | Prime850