Disallow indexing of ALL subdomains
-
I'm using www.domain.com as my development hosting. Each website that i'm developing get's a temporary URL like this:
project1.domain.com
project2.domain.com
project3.domain.com
...Now i'd like to set that ALL these subdomains can not be indexed in Google. Now I manually have to do this for each subdomain's site, and when I go online I have to change the robots.txt again. So I would like to make things a bit easier for me.
Is this possible?
-
Hello there!
Like Wesley mentioned before the best way to avoid any "non-desired" crawling and indexing of your development or testing environment is by requiring authentication (whether with htaccess or your own programmed login/password screen).
Unfortunately sometimes there have been situations when search crawlers don't necessarily follow robots.txt directives. Additionally, beyond search engines you might want to protect your development or testing environment, making sure that only people with the required access to it can enter. Because of this, the best way to go is requiring authentication to access to your subdomains, not robots.txt.
I hope this help!
-
Nobody?
-
Thanks for the link, but I don't think that will solve my problem?
When Google wants to crawl project1.domain.com, he won't check the .htaccess of domain.com right? So then he'll just be crawling that subdomain?
-
You should edit your .htaccess file as is described here:
http://stackoverflow.com/questions/6738896/excluding-testing-subdomain-from-being-crawled-by-search-engines-w-svn-repositI hope i answered your question
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
titles length, URL length and meta descriptions on a subdomain effecting SEO on main domain?
Hi all, I am currently evaluating areas for optimization on my main domain. When doing this, Moz has identified multiple titles and urls that should be shortened and missing meta descriptions on my subdomain (a help center of sorts). As far as I am aware, we have not set up any "no-index" rules for this subdomain. Are these items affecting SEO on my main domain? Thanks,
On-Page Optimization | | annegretwidmer
Kasey0 -
Meta Titles On Site Different From Google Index Page
This is very embarrassing but I hope someone can help. The Meta Titles on my site are not being shown correctly on the Google site index. For example, when I got directly to the page the appropriate page title is shown obviously. However when I go view that page on Google, the title in completely different. Page title: <a class="attribute-value">Web Design, SEO, PPC, Mobile Development In Philadelphia, Bucks| Infinity Digital Agency</a> Google Shows: Our Services - Infinity Digital Agency Here is the result page. I am currently running WordPress and Yoast. Any thoughts would be greatly appreciated. https://www.google.com/search?q=site%3Awww.infinitydigitalagency.com%2Fservices%2F&ie=utf-8&oe=utf-8
On-Page Optimization | | infinitydigitalagency0 -
Index dropped 20 pages at once since yesterday
Hi community, I just realized that my indexed pages dropped from the amount of 95 to 75 and I don't know why. I did some title tag arrangements because we are launching with our first product (before that it was just a blog). I did these changes 1 week ago and fetched to google the homepage and some subdomains. Thanks for your help. Kind regards Marco
On-Page Optimization | | Marc19870 -
Index / Monthly Click Number
Hi,
On-Page Optimization | | HypermediaSystems
This is a general question, so sorry in advance if inappropriate. Once I was told, in large scale EC / Forum Site,
the following number should be around 1,
and if it is below 1, it is a good sign ... Google Indexed Page Number / Monthly ( 30days ) Click Number I was told this is just a general idea, and real world situation varies, then
if you don't have any standard, this could be a start. (not dogmatic rules, just reference) Does this sounds about right? or do you have any other formula? I was tasked to do the site wide SEO, and diagnose the general state of SEO-wellness/fitness..Â
and right now, the number is 1.5, so I am about to report we can do more to get more SERP presence or something... If you guys point me relevant blog article / Q&A forum, I would really appreciate. Thanks!0 -
Google indexing page differently
Does google index an interal page differently depending on whether you are using a FULL url (including domain) or just a relative link? Â Also, is it possible that using a full URL (http://mysite.com/page.html) causes the browser to "ping" the server differently than just having the href linked to using relative links (/page.html) Could this cause server or firewall perfomance issues?
On-Page Optimization | | WebRiverGroup0 -
Sitemap error is reported when using a sitemap-index generated by Yoast
I've installed the Yoast SEO Plugin for wordpress and I've setup the sitemaps using it. I saw the tool has generated the Sitemap index file http://www.phraseexpander.com/sitemap_index.xml with different indexes for posts and pages I've submitted that to google and it's indexed. When I use Seoquake to check my website, I see that it says that the sitemap is missing (in fact http://www.phraseexpander.com/sitemap.xml) is returning 404. Shall I fix that? Shall I do a 301 redirect in my .htaccess file to http://www.phraseexpander.com/sitemap_index.xml Thanks.
On-Page Optimization | | nagar0 -
Does Google index dynamically generated content/headers, etc.?
To avoid dupe content, we are moving away from a model where we have 30,000 pages, each with a separate URL that looks like /prices/<product-name>/<city><state>,  often with dupe content because the product overlaps from city to city, and it's hard to keep 30,000 pages unique, where sometimes the only distinction is the price & the city/state.</state></city></product-name> We are moving to a model with around 300 unique pages, where some of the info that used to be in the url will move to the page itself (headers, etc.) to cut down on  dupe content on those unique 300 pages. My question is this. If we have 300 unique-content pages with unique URL's, and we then put some dynamic info (year, city, state) into the page itself, will Google index this dynamic content? The question behind this one is, how do we continue to rank for searches for that product in the city-state being searched without having that info in the URL? Any best practices we should know about?
On-Page Optimization | | editabletext0 -
Which is Best Practice for creating URLs for subdomain?
My website is related to education. We have created sub domains for all major colleges, universities & Entrance exams like Gre, Toefl ETC. for eg: Â amityuniversity.abc.com (Amity is Name of University ) Now if have to mention city name in URL as well (college is located in multiple locations) amityuniversity-delhi.abc.com
On-Page Optimization | | rohanarora536
amityuniversitydelhi.abc.com Now my Q is can we use hyphens in sub domains if we have to add city name or shall we create without using any  hyphens. In Directory  structure we can always separate words with hyphens, can we follow same practice in subdomain as well Which is a  best URL for subdomain amity-university-delhi.abc.com
amityuniversity-delhi.abc.com
or amityuniversitydelhi.abc.com0