Need only tens of pages to be indexed out of hundreds: Robots.txt is Okay for Google to proceed with?
-
Hi all,
We 2 sub domains with hundreds of pages where we need only 50 pages to get indexed which are important. Unfortunately the CMS of these sub domains is very old and not supporting "noindex" tag to be deployed on page level. So we are planning to block the entire sites from robots.txt and allow the 50 pages needed. But we are not sure if this is the right approach as Google been suggesting to depend mostly on "noindex" than robots.txt. Please suggest whether we can proceed with robots.txt file.
Thanks
-
Hi vtmoz,
Given the limitations you are telling us, I'd give noindex in robots.txt a try.
I've run some experiments and found that noindex rule in Robots.txt works. It definitely won´t remove from index that pages, but it will stop showing them for search results.I'd suggest you to try using that rule with care.
Also, run some experiments on your own. My first test would be only adding one or two pages, the one that causes more trouble being indexed (maybe due to undesired traffic or due to ranking on undesired search terms).Hope it helps.
Best luck!
GR
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Algo Updates
Hi I have noticed strange drops in our rankings, some last Monday 12th & some Sunday 18th - has anyone else experienced anything on these dates? I've seen talk of a possible algo update, but nothing concrete Thanks!
Algorithm Updates | | BeckyKey0 -
What Ranking Factors Impact Google News Visibility?
I'm just at the beginning of a new analysis involving Google News visibility and ranking factors, and thought I could put the project out to you, dear SEO geniuses, to get your ideas and perspectives. Backgrounder: My company operates over 50 niche, disease-specific daily news sites, covering science, research and advocacy news about specific diseases. Virtually all of them are in Google News. They range in age from 3 years old to 3 months old. Varying degrees of page rank / authority Content on the site is completely niche to specific diseases, and we have a lot of sites for rare and orphan diseases. Most of the content is news, but we also have info/resources pages, blogs, and some short-form posts made for use in social media. The Project: I want to do an analysis of keywords in our news headlines and see how certain keywords correlate with articles that do well -- both in terms of search traffic and overall with users. Going to use our Multiple Sclerosis News Today website. Most of our search traffic comes from Google News. What I hope to gain: I'm curious to see if certain sets of keywords that relate to the disease, to therapies, etc. drive the most traffic. I want to compare these keyword lists to how well we rank in organic search for the same keywords (via news articles or info pages) to see if there is a connection. I want to also create a working keyword list of the best-performing keywords in the news as a way of cross-pollinating content production on our blogs, info pages, social content, etc. I want to increase my knowledge base of ranking factors specific for Google News. The last point is really something I wish I knew more about. I feel like there aren't many knowledge resources out there about Google News. Is it safe to assume that the same on-site and off-site SEO best practices that govern organic search engine visibility are at play in Google News, or are there independent factors as well? I'd love to get your thoughts. Thanks!
Algorithm Updates | | Michael_Nace0 -
Delay between being indexed and ranking for new pages.
I've noticed with the last few pages i've built that there's a delay between them being indexed and them actually ranking. Anyone else finding that? And why is it like that? Not much of an issue as they tend to pop up after a week or so, but I am curious. Isaac.
Algorithm Updates | | isaac6630 -
Best Moz article on landing pages?
From what I understand, building landing pages to link back to sites is a thing of the past. I am looking for a good article that explains best current landing page practices (post Panda and Penquin). Any suggestions?
Algorithm Updates | | cschwartzel0 -
Google Maps marker inconsistency
We just discovered that depending on the address format you enter into Google, you may come across incorrectly placed marker locations on Google Maps. Is this because our Google Places address format is not consistent with Google Maps' format? If so, when I go into Google Places to update the address format, am I going to have to go through the citation process all over again?
Algorithm Updates | | SSFCU0 -
Drop in Page Indexing, Small rise in Search Queries
Hello, I have a news based website so i am creating multiple new posts daily. I changed a lot of the site and got rid of old potentially duplicate content back in feb and had a sharp drop in pages indexed. I know this was because I removed a lot of pages though. However I still have a good 20,000 + pages on my site and my indexing has dropped a further three times since then. From 9,000 to 2,000 a coupe of months ago and then slowly down since April to just 133. It doesn't seem to have affected my search queries yet but surely will if it continues. I am really confused as to how this might happen & how to turn it around. We dont use any dodgy SEO tricks either.
Algorithm Updates | | luwhosjack0 -
Page details in Google Search
I noticed this morning a drop in the SERPs for a couple of my main keywords. And even though this is a little annoying the more pressing matter is that Google is not displaying the meta title I have specified for the majority of my sites pages, despite one being specified and knowing my site has them in place. Could this sudden change to not using my specified title be the cause of the drop, and why would they be being displayed by Google in the first place, when they are there to be used. The title currently being displayed inthe SERPs is not anything that has been specified in the past or from the previous latest crawl etc. Any insight would be appreciated. Tim
Algorithm Updates | | TimHolmes0 -
How To Rank High In Google Places?
Hello SEOmoz, This question has been hounding me for a long time and I've never seen a single reliable information from the web that answers it. Anyway here's my question; Supposing that there are three Google places for three different websites having the same categories and almost same keywords and same district/city/IP how does Google rank one high from the other? Or simply put if you own one of those websites and you would want to rank higher over your competitors in Google places Search results how does one do it? A number of theories were brought up by some of my colleagues: 1. The age of the listing 2. The number of links pointing to the listing (supposing that one can build links to ones listing) 3. The name/url of the listing, tags, description, etc. 4. The address of the listing. 5. Authority of the domain (linked website) You see some listings have either no description, and only one category and yet they rank number one for a specific term/keyword whereas others have complete categories, descriptions etc. If you could please give me a definite answer I will surely appreciate it. Thank you very much and more power!
Algorithm Updates | | LeeAnn300