Blocking Subdomain from Google Crawl and Index
-
Hey everybody, how is it going?
I have a simple question, that i need answered.
I have a main domain, lets call it domain.com. Recently our company will launch a series of promotions for which we will use cname subdomains, i.e try.domain.com, or buy.domain.com. They will serve a commercial objective, nothing more.
What is the best way to block such domains from being indexed in Google, also from counting as a subdomain from the domain.com. Robots.txt, No-follow, etc?
Hope to hear from you,
Best Regards,
-
Hello George, Thank you for fast answer! I read that article and there is some issue with that. if you can see at it, i'd really appreciate it. So the problem is that if i do it directly from Tumblr, it will also block it from Tumblr users. Here is the note right below that option "Allow this blog to appear in search results":
"This applies to searches on Tumblr as well as external search engines, like Google or Yahoo."Also, if i do it from GWT, i'm very concerned to remove URLs with my subdomain because i afraid it will remove all my domain. For example, my domain is abc.com and the Tumblr blog is setup on tumblr.abc.com. So i afraid if i remove tumblr.abc.com from index, it will also remove my abc.com. Please let me know what you think.
Thank you!
-
Hi Marina,
If I understand your question correctly, you just don't want your Tumblr blog to be indexed by Google. In which case these steps will help: http://yourbusiness.azcentral.com/keep-tumblr-off-google-3061.html
Regards,
George
-
Hi guys, I read your conversation. I have similar issue but my situation is slightly different. I'll really appreciate if you can help with this. So i have also a subdomain that i don't want to be indexed by Google. However, that subdomain is not in my control. I mean, i created subdomain on my hosting but it is pointing to my Tumblr blog. So i don't have access to its robot txt. So can anybody advise what can i do in this situation to noindex that subdomain?
Thanks
-
Personally I wouldn't rely just on robots.txt, as one accidental, public link to any of the pages (easier than you may think!) will result in Google indexing that subdomain page (it just won't be followed). This means that the page can get "stuck" in Google's index and to resolve it you would need to remove it using WMT (instructions here). If there were a lot of pages accidentally indexed, you would need to remove the robots.txt restriction so Google can crawl it, and put a noindex/nofollow tags on the page so Google drops it from its index.
To cut a long story short, I would do both Steps 1 and 2 outlined by Federico if you want to sleep easy at night :).
George
-
It would also be smart to add the subdomains in Webmaster Tools in case one does get indexed and you need to remove it.
-
Robots.txt is easiest and quickest way. As a back up you can use the Noindex meta tag on the pages in the subdomain
-
2 ways to do it with different effects:
-
Robots.txt in each subdomain. This will entirely block any search engine to even access those pages, so they won't know what they have inside.
User-Agent:*
Disallow: /
-
noindex tags in those pages. This method allows crawlers to read the page and maybe index (if you set a "follow") the pages to which you link to.or "nofollow" if you don't want the linked pages to be indexed either.
Hope that helps!
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Crawl Shows Duplicate Content Which Doesn't Seem To Appear In Google?
Morning All, First post, be gentle! So I had Moz crawl our website with 2500 high priority issues of duplicate content, not good. However if I just do a simple site:www.myurl.com in Google, I cannot see these duplicate pages....very odd. Here is an example....
On-Page Optimization | | scottiedog
http://goo.gl/GXTE0I
http://goo.gl/dcAqdU So the same page has a different URL, Moz brings this up as an issue, I would agree with that. However if I google both URL's in Google, they will both bring up the same page but with the original URL of http://goo.gl/zDzI7j ...in other words, two different URL's bring up the same indexed page in Google....weird I thought about using a wildcard in the robots.txt to disallow these duplicate pages with poor URL's....something like.... Disallow: /*display.php?product_id However, I read various posts that it might not help our issues? Don't want to make things worse. On another note, my colleague paid for a "SEO service" and they just dumped 1000's of back-links to our website, of course that's come back to bite us in the behind. Anyone have any recommendations for a good service to remove these back-links? Thanks in advance!!0 -
Should I "No Index" Certain Pages On My Site?
I have some pages on my site that don't really have any content other than some iframes that are embedded from another site. I thought it would be best to tag the page with a no-index so that search engines would leave the page alone since it has no content as far as the search engine can tell (but does provide value to my site visitors). Is this the proper approach or does it do more harm than good?
On-Page Optimization | | Kyle Eaves0 -
Duplicate Page Titles in Crawl Errors (although Google is rewriting in serps ??)
Hi Im working on a client/project and crawl report is showing thousands of dupe page titles In the case of the blog/news section its aprox 50 since aprox 50 posts and they all have the same meta-title: "Brand News | Brand" as opposed to: "Title Unique to Page/Topic/KW Relating to Content | Brand" Since these are the main content pages we want to rank (in addition to the main site category pages) then i have instructed dev must prioritise populating these pages meta-titles with the actual post/article titles, as per the latter version of the above example. (I should mention that i have requested they fix all dupe titles but main content pages are the priority). Whilst this will reduce the number of dupe titles in crawl error/warning report which is a good thing, is it actually likely to increase the ranking of these news/content pages given that Google does seem to be rewriting the titles correctly in the serps based on the page content ? Many Thanks in advance for your input
On-Page Optimization | | Dan-Lawrence0 -
Google Drop
I started using SEOMOZ due to a sudden and huge drop in Google for two main keywords (hair bows and baby headbands). Our site (BloomingBows.com) has held a top three spot for years with these words and then in the last few months has dropped down on the first page and now they are completely off the charts. Is there any insight as to why? Also, we have been very active using the data from here in the last week or so to clean up and improve anything listed, but I am still seeing keywords drop into the 40 - 60 position and our traffic is drying up. Starting to panic and wondering if I am missing something or going about this in the wrong way. ANY insight is appreciated at this point!! Thank you!!
On-Page Optimization | | bloomingB0 -
How to properly remove pages and a category from Google's index
I want to remove this category http://www.webdesign.org/web-design-news-all/ and all the pages in that category (e.g. http://www.webdesign.org/web-design-news-all/7386.html ) from Google's index. I used the following string in the "Reomval URS" section in Google Webmaster Tools: http://www.webdesign.org/web-design-news-all/* is that correct or I better use http://www.webdesign.org/web-design-news-all/ ? Thanks in advance.
On-Page Optimization | | VinceWicks0 -
My Images Aren't Indexed By Goolge
My site is indexed by google, and there hasn't really been much of a problem with my content, but now I am noticing that none of our images are coming up in the images searches of google. I mean none. I have even typed in the alt text of the image verbatim and nothing shows up. I use wordpress if this helps anyone. Any advice would be awesome, Thanks a lot.
On-Page Optimization | | Caseman0 -
Possible Reasons for 40% Drop in Google since January?
Hello: I know this is long - but I kept it as succinct as I could to explain the situation - but everyone of the brilliant people on here insight is so greatly appreciated in advance. There has been more than a 30% decline in traffic mostly from Google on a site with an 11 year presence on the web, 5 page rank (historically 6), fairly good quality backlinks (but they are old). I cannot pinpoint a smoking gun (such as a specific penalty) - if it's just not attrition in demand for the product itself which there is plenty of evidence for. But I practice only white hat, so I don't know what it is. I do not engage right now in link building because my time to work on SEO is very limited. We have a few common keywords that cannot be avoided (because there are no good synonyms) so there is a high density for one word especially all over the site. From some websites (like tidbits) we have in Google Webmaster account tens of thousands of backlinks. How serious is this (FYI - they're not new though) Because of some unflattering reviews from sites with strong authority, the number of click throughs vs. impression in Google Search has dropped dramatically - I think a big part of it is that because the reviews drove down CT ratio. Interestingly, the number of CTs from Adwords has also dropped off significantly, and I was told by Adwords specialist that declining organic traffic does affect the Adwords traffic to - in fact, they're not even using our daily budget for Adwords as they used to. The site also doesn't pick up quality organic backlinks as it used to, and the anchor link text is almost always the same - a product name. Plus, the site was banned in 2008 by Google (from a black hat keyword stuffing that I discovered happened in another dept.) and only after really working hard was it reinstated by reconsideration request. Site traffic from Google has never fully reached it's potential since then. Also, text prices were removed from Buy Now buttons even though the prices were textually part of the buttons for years (could this be some penalty?) **Is it possible that Google is taking the ban history, the poor reviews, the decline in CTs to impressions on the site, the huge decline (75% over four years) in "search volume" or "search interest" (according to Google Trends) for the site's main product name and just using all of this to keep downgrading the site? If true, wouldn't the page rank be diving down to anywhere from 0 to 2 (or is that not relevant at all?) ** There are no messages of a penalty or anything else in the official Webmaster account for this site. Any insight anyone is able to provide is much appreciated.
On-Page Optimization | | holdtheonion1 -
Why some pages are not indexed?
I have a furniture´s ecommerce. When searching for "site: movstore.com.br" returned 1080 results, but if I search for "site: movstore.com.br / Product" returned 1020 results. I mean, that 1080 indexed pages, 1020 are products pages and the other 60 pages are irrelevant. Where are the category pages? "site: movstore.com.br / Categories" - 0 results
On-Page Optimization | | maisempresas
"site: movstore.com.br / Departments" - 0 results
"site: movstore.com.br / Marks" - 0 results What might be happening?0