Disallow indexing of ALL subdomains
-
I'm using www.domain.com as my development hosting. Each website that i'm developing get's a temporary URL like this:
project1.domain.com
project2.domain.com
project3.domain.com
...Now i'd like to set that ALL these subdomains can not be indexed in Google. Now I manually have to do this for each subdomain's site, and when I go online I have to change the robots.txt again. So I would like to make things a bit easier for me.
Is this possible?
-
Hello there!
Like Wesley mentioned before the best way to avoid any "non-desired" crawling and indexing of your development or testing environment is by requiring authentication (whether with htaccess or your own programmed login/password screen).
Unfortunately sometimes there have been situations when search crawlers don't necessarily follow robots.txt directives. Additionally, beyond search engines you might want to protect your development or testing environment, making sure that only people with the required access to it can enter. Because of this, the best way to go is requiring authentication to access to your subdomains, not robots.txt.
I hope this help!
-
Nobody?
-
Thanks for the link, but I don't think that will solve my problem?
When Google wants to crawl project1.domain.com, he won't check the .htaccess of domain.com right? So then he'll just be crawling that subdomain?
-
You should edit your .htaccess file as is described here:
http://stackoverflow.com/questions/6738896/excluding-testing-subdomain-from-being-crawled-by-search-engines-w-svn-repositI hope i answered your question
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to fix duplicate content for homepage and index.html
Hello, I know this probably gets asked quite a lot but I haven't found a recent post about this in 2018 on Moz Q&A, so I thought I would check in and see what the best route/solution for this issue might be. I'm always really worried about making any (potentially bad/wrong) changes to the site, as it's my livelihood, so I'm hoping someone can point me in the right direction. Moz, SEMRush and several other SEO tools are all reporting that I have duplicate content for my homepage and index.html (same identical page). According to Moz, my homepage (without index.html) has PA 29 and index.html has PA 15. They are both showing Status 200. I read that you can either do a 301 redirect or add rel=canonical I currently have a 301 setup for my http to https page and don't have any rel=canonical added to the site/page. What is the best and safest way to get rid of duplicate content and merge the my non index and index.html homepages together these days? I read that both 301 and canonical pass on link juice but I don't know what the best route for me is given what I said above. Thank you for reading, any input is greatly appreciated!
On-Page Optimization | | dreservices0 -
What is the fastest way to re-index an important page?
Hello Moz Community Members, Besides submitting the URL in Google Webmaster, what are other ways to make sure google indexes/crawls a page which was noindexed?
On-Page Optimization | | SEMEnthusiast0 -
Latent semantic Indexing - Does this help rankings/relevance?
Hi, Does semantically related words to the target term on a page help with rankings/relevance? If your after the term 'PC Screen' and you use the term 'PC Monitor' will go make the connection and also reward you because of the relevance? Anyone do this and have you seen any positives? I've just started to try this out lately and have been combining it with Wordle.net to give me an indication of where the content piece is heading and how aggressive the content leans towards certain words (makes things a little more interesting then calculating densities).
On-Page Optimization | | Bondara0 -
Subdomain vs. Root Domain design
I know that there are penalties if you have a navigation item in your categories that links to another site (root domain) that yields the same design as the original link. Is there any negative SEO implications with having a subdomain with the same design as the root domain i.e. root.com vs. store.root.com And the navigation functionality within will have links to both store.root.com as well as root.com directories? Thanks in advance.
On-Page Optimization | | customerparadigm.com0 -
Google Index Report
Hi, I have just checked my google webmaster tools account and viewed the index status of my website and it produced the attached graph, which show quite a big spike in indexing during July and August 2012. Does this look normal or does it reveal anything peculiar? We did have a new website launched in June 2012 and I re-submitted the sites URL's to google as part of the re-launch and so I am unsure if this may account for the spike. Any advice appreciated. Thanks indexing.png
On-Page Optimization | | UnderMe0 -
Will Google re-index the page if content is changed/improved?
Hello,
On-Page Optimization | | wickedsunny1
i have a little question, it seems most of my posts which are 3 years old or even more, all have almost same type of 2-3 lines of text at post starting and then images roundups etc.Â
which recently stopped getting any traffic from google. Probably becuase of lack in text etc.Â
So if i now edit those old pages, will they be reindexed in google with this new content/data?Â
thanks in advance.
cheers0 -
How long does it take Google to index new title tags and meta descriptions?
Hi, I have launched a website ( www.bookkeepingking.com )and would like Google to reflect the latest title tag changes in its results. How long does it typically take to happen? I believe this might be one reason why the site may not be generating any search traffic yet, even though I haven't yet begun any link building yet. A couple of notes: When I enter site:bookkeepingking.com in the search box, the pages show up, but with the old/default title and meta descriptions. I am using the Yoast for SEO plugin I have resubmitted the XML map - Yoast created 2... /page-sitemap.xml and /post-sitemap.xml The changes were made about 1 week ago. WMT says that there are zero errors. Is there something I may have missed? Or do I just need to give it more time? Thanks in advance for your help. Erin
On-Page Optimization | | HiddenPeak0 -
Google indexing Internal Search Results
Greeting, Currently I have noticed that Google is starting to index our internal search page results. Should I block those pages in our robot txt file or have you ever heard of any websites that actually gained traffic or rank by letting Google index those pages? Thanks
On-Page Optimization | | Tonyd230