Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Spanish website indexed in English, redirect to spanish or english version if i do a new website design?
Hi MOZ users, i have this problem. We have a website in Spanish Language but Google crawls it on English (it is not important the reasons). We re made the entire website and now we are planning the move. The new website will have different language versions, english, spanish and portuguese. Somebody tells me that we have to redirect the old urls (crawled on english) to the new english versions, not to the spanish (the real language of the firsts). Example: URL1 Language: Spanish - Crawled on English --> redirect to Language English version. the other option will be redirect to the spanish new version, which the visitor is waiting to find. URL1 Language: Spanish - Crawled on English --> redirect to Language Spanish version. What do you think? Which is the better option?
Web Design | | NachoRetta0 -
Best practice for multilanguage website ( PHP feature based on Browser or Geolocalisation)
Hi Moz Experts I would like to know what does it the best practice for multilanguage website for the default language ? There are several PHP features to help users to get the right language when they come from SEO and direct; present the default language by browser language, by gelolocalisation, etc. However, which one is the most appropriate for Quebec company that try to get outside Canada ? PRO and CONS. Thank you in advance.
Web Design | | johncurlee0 -
Privacy Policy: index it/? And where to place it?
Hi Everyone, Two questions, first: should you allow google to index your privacy policy? Second: for a service based site (not e-commerce, not selling anything) should you put the policy in the footer so it's site wide or just on the "contact us" form page? Best, Ruben
Web Design | | KempRugeLawGroup0 -
Duplicate content on websites for multiple countries
I have a client who has a website for their U.S. based customers. They are currently adding a Canadian dealer and would like a second website with much of the same info as their current website, but with Canadian contact info etc. What is the best way to do this without creating duplicate content that will get us penalized? If we create a website at ABCcompany.com and ABCCompany.ca or something like that, will that get us around the duplicate content penalty?
Web Design | | InvoqMarketing0 -
Website Redesign SEO Checklist
I am looking for a comprehensive Website Redesign SEO Checklist. One of my customers is changing its website design (including URL structure) and I need all kinds of suggestions for a smooth SEO operation. Thanks in advance!
Web Design | | stradiji0 -
Is it possible to redirect the main www. domain - but keep a subdomain active?
Hi Mozzers, Quick question, which I hope one of you can answer... Let's say I have a website (i) www.example.com and on that a subdomain exists, (ii) subdomain.example.com. Let's say I want to change my main domain from www.example.com to www.newwebsite.com. I'd 301 all content, use GWT to notify Google of a change of address etc etc. Having done that, is it still possible to keep the original subdomain active? So, even though www.example.com has been redirected / transferred to www.newwebsite.com, subdomain.example.com would still exist. If that is possible, what is the implication for Domain Authority? On the one hand, I have transferred the main site (so DA from that will transfer to the new site); but part of that root domain is still active. Make sense? Any answers? Thanks everyone...
Web Design | | edlondon0 -
3 Brands, 3 Services, 3 Different Websites Right?
My client was told that having 1 website for 3 different brands/services is better than having 3 websites. I need your help to prove my value to a new client. This client has worked with Reach Local on PPC for some time and when they first got started the Reach Local Markering Consultant told this cleint that they needed to have one site for better SEO purposes. The client was told that Google ranks websites higher if they have more paid traffic going to them. I've been doing this for long enough to realize this does not help ranking, at least not enough to make a difference. Keep in mind this is for 3 different companies. One company does plumbing, another electrical and the last one does air conditioning. They also have 4 locations but only two locations have mutliple services opperating out of them. I understand these 2 location will not have there own Google+ Local / Places listing. Using the same address for 2 different business and expecting a first page ranking is just not possible. Right now when you visit the clients website you see a logo that rotates with a banner section that follows the logo rotation. First you see the AC Company and then the Plumbing etc. I see this as confusing to the end user and it is more work to get it ranked for SEO. I recommended that we build 3 speerate websites for each service and just list out all the addresses that the company services on the contact page. I would also design inside the footer links to the other services for branding purposes. Please share your thoughts on how you would handle this if you were doing the SEO for your own 3 different business services. I really appreicate any input/insight to this. Thank you so much in advance!!!!
Web Design | | 1SMG0 -
Site-wide footer links or single "website credits" page?
I see that you have already answered this question before back in 2007 (http://www.seomoz.org/qa/view/2163), but wanted to ask your current opinion on the same question: Should I add a site-wide footer link to my client websites pointing to my website, or should I create a "website credits" page on my clients site, add this to the footer and then link from within this page out to my website?
Web Design | | eseyo0