Disallow indexing of ALL subdomains
-
I'm using www.domain.com as my development hosting. Each website that i'm developing get's a temporary URL like this:
project1.domain.com
project2.domain.com
project3.domain.com
...Now i'd like to set that ALL these subdomains can not be indexed in Google. Now I manually have to do this for each subdomain's site, and when I go online I have to change the robots.txt again. So I would like to make things a bit easier for me.
Is this possible?
-
Hello there!
Like Wesley mentioned before the best way to avoid any "non-desired" crawling and indexing of your development or testing environment is by requiring authentication (whether with htaccess or your own programmed login/password screen).
Unfortunately sometimes there have been situations when search crawlers don't necessarily follow robots.txt directives. Additionally, beyond search engines you might want to protect your development or testing environment, making sure that only people with the required access to it can enter. Because of this, the best way to go is requiring authentication to access to your subdomains, not robots.txt.
I hope this help!
-
Nobody?
-
Thanks for the link, but I don't think that will solve my problem?
When Google wants to crawl project1.domain.com, he won't check the .htaccess of domain.com right? So then he'll just be crawling that subdomain?
-
You should edit your .htaccess file as is described here:
http://stackoverflow.com/questions/6738896/excluding-testing-subdomain-from-being-crawled-by-search-engines-w-svn-repositI hope i answered your question
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to block index of link and content
Hi, We have pages where articles are shown and in the sides we have small snippets of Articles which shows the title and close to 25 words and a image. When i search for something in Google the snippet image and content is shown and in Google when clicked it redirects to a page which is not meant to be shown for the keyword the visitor is querying Is there a way i can block all the links and content shown in the right and left side of the page so Google does not get confused with the page content thats not related to that page? thanks
On-Page Optimization | | AlexisWithers0 -
Using Canonical & INDEX, FOLLOW meta tag
I am sort of new to this, and working with an overseas development team on an eCommerce site. I am curious if the Canonical tag will supersede the tag, or if it is best to remove this tag entirely for duplicate product pages?
On-Page Optimization | | BretDarby1 -
Paginated URLs are getting Indexed
Hi, For ex: - My site is www.abc.com and Its paginated URLs for www.abc.com/jobs-in-delhi are in the format of : www.abc.com/jobs-in-delhi-1, www.abc.com/jobs-in-delhi-2 and vice versa also i have used pagination tags rel=next and rel=prev. My concern is all the paginated URLs are getting indexed so is their any disadvantage if these URLs are getting indexed as somewhere i have read that link juice may get distributed in case of pagination. isn't it good to use Noindex, Follow so that we can make the Google to understand that paginated page are not so much important and that should not be ranked.
On-Page Optimization | | vivekrathore0 -
Does Widgetised Content Index The Same As A Regular Page
Hi, We have a website that was built in my opinion bizarrely where the bottom half of the page where most of the content is, is a widget. I just wondered if the content being in a widget is indexed any differently. I ask as normal pages seem to index and rank much better than the wordpress template using the widget. Hope someone might be able to clarify this. Thanks
On-Page Optimization | | denismilton0 -
Title Tags for Index Pages
What tactics do you use to change the title tags of your index page so they're not all the same? For example, if you have an index page that has 100 pages, each with the same title, what tactics do you use to give each page a unique title and how important is it?
On-Page Optimization | | felt0 -
Why Aren’t All My XML Sitemap Images Indexed in Webmaster Tools?
Hi, Here is our main sitemap http://www.vistastores.com/newsitemap/main_sitemap.xml We have submitted all category wise sitemap having Image Tags : For eg -Â Ac Category http://www.vistastores.com/newsitemap/window_ac_sitemap.xml contains iamge tag - image:imageimage:locimage:captionimage:title</image:title></image:caption></image:loc></image:image> All our 142 category pages includes these format. Still the sitemap report on 4-Apr-2013 says: Sitemaps content Web pages:
On-Page Optimization | | CommercePundit
Submitted   14,569
Indexed      11,219 Images:
Submitted   21,442  Â
Indexed      11,762 You can see major difference in submitted v/s indexed. I have looked into Jay Simpson question - http://www.seomoz.org/q/any-idea-why-our-sitemap-images-aren-t-indexed to find this answer but didn't get Perfect & clear answer. I need urgent answer to fix this issue..... K0NDuw5s.jpg0 -
0 urls indexed in GWT, many found with site: command
Hi, This is happening with a brand new site. We have created sitemaps and submitted them to Google Webmaster Tools. GWT says sitemaps are ok, "x" number of urls submitted, but no urls indexed. When I check in Google with site:domain.com I see that many of the urls are already indexed. Why this discrepancy between GWT and reality? Thanks for your time!
On-Page Optimization | | gerardoH0 -
Got loads of pages, but none indexing?
I have a WordPress site with loads of pages on a url like this http://mysite.com.au However, Google has indexed http://www.mysite.com.au and as a result only indexing 2 pages. Â How do I fix this? Many thanks Dan
On-Page Optimization | | Pokodot0
