How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best Practices for home page design for ecommerce website
I know this question is not directly related to SEO, but I figured I have been getting some good help from this forum, so why not? The website is www.vrtack.com. I am looking to redesign the home page. It is an ecommerce website selling equestrian clothing and leather goods. My goals are: 1. Reduce the very high bounce rate and drop-off rate. 2. Fine tune the relevancy of the website towards a handful of keyword phrases. 3. Engage the visitor to create better click-through and to increase the average time spent on the page/site. 4. Page Loading time is of importance. It has to load quickly. I would love to hear some specific suggestions, examples, best practices.
Web Design | | amitramani0 -
Redesigning a really old Website with old-fashioned permalinks
Hey SEO-Pros, I'm currently redesigning a quite old website, with the following URL structure: TLD/category/category.php?interview_id=819 The new Version will be a little more SEO-friendly: TLD/interviews/name-of-interview/ I know I have to do a 301-Redirect for all the old URLs to the new ones in order to keep the (until now pretty good) Google Ranking. If the rankings drop after the redesign has been done, I'll surely get killed 😉 So, Is there any easy way of creating these 301-Redirects (must be thousands of URLs...)? Best Regards guys and thanks for your help!
Web Design | | dominator0 -
Web Developer Using Stock Photos
Hello, The organization is selling a cms system in a niche market across the country. It has the normal SEO challenges, in addition he is using purchased stock images. This seemed ok, while he was smaller but now we are growing rapidly and these images are VERY STOCK- and well used ( I have checked with Tiny Eye). I remember a few years ago this was a flag to the search engines who went through manual review, is this still true? It seems to me that the theme's that come with the images, are duplicated ( including navigation & footers), so having the duplicated images would be another negative. Thank you for your suggestions!
Web Design | | TammyWood0 -
New website put up and ALL my keywords fell a LOT!???
I helped a client redesign their new website and we just went live a couple weeks ago. This morning I checked his campaign and 53 keywords fell DRAMATICALLY. Like 35-50 places down in Google for dozens of keywords!? I haven't ever seen a drop that's so dramatic when putting up a new site. Have you ever seen this? Will they bounce back? This site isn't significantly different than the last one. We did forward two other domains to this new site but that wouldn't make a difference, would it? Any feedback would be greatly appreciated! Matthew
Web Design | | Mrupp440 -
What do YOU look for when reviewing technical website criteria?
Wanted to get everyone's thoughts on this. One of my clients was asking me what they thought of a certain website design company. They wanted to use this company because their designs were very beautiful but wanted to be sure that contracting them would be the right decision. After I made my formal review and recommendation I thought this might be a great topic of discussion for the amazing SEOMOZ community. So, what factors do you look at when determining which design firms are worth their weight?
Web Design | | JusinDuff0 -
Map Search Tools to integrate on Accommodation Website
Hello all, Can any recommend a Map Search tool that i can integrate into my accommodation website. Ideally I want to be able to pin my clients on the map search with links back to their listing page on my website and provide an alternative search facility for clients looking for accommodation. Am covering the South Africa region specifically. Am assuming that i could go down the Google Maps route but would really like to know what alternatives there are on offer. Also what SEO considerations do i need to think about when adding this to my website. Thanks in advance for any help.
Web Design | | SamanthaRiggien0 -
Does File Compression software on a website benefit SEO?
Hi all. Should we be using File Compression software on our website files for SEO benefit? If so, do you like Deflate or gzip? Thank you for your help! Jay
Web Design | | theideapeople0 -
Old links in Google, new website affecting SEO?
Hi Guys, I have launched my website in october and it has already been indexed by google. Now I'm going to launch my redesign which comes with a new structure, content, links, etc. So the question is, do I have to resubmit my website to google to get rid of old links? Onsite Explorer shows links to my forum which has been spammed with p* stuff which has been already indexed as well. The forum is off now. I want to use SEOmoz to track my new website but I guess this could be a hard thing as old links etc will be shown as well. Is there any tool to let Google know about my changes? Does it affect my SEO in any way? Thank you for your help. Nick
Web Design | | NickITW0