How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What type of website is best for seo.
I need a new website for my health insurance business. What type is best for SEO? Many thanks
Web Design | | laurentjb0 -
When rel canonical tag used, which page does Google considers for ranking and indexing? A/B test scenario!
Hi Moz community, We have redesigned our website and launched for A/B testing using canonical tags from old website to new website pages, so there will be no duplicate content issues and new website will be shown to the half of the website visitors successfully to calculate the metrics. However I wonder how actually Google considers it? Which pages Google will crawl and index to consider for ranking? Please share your views on this for better optimisation. Thanks
Web Design | | vtmoz0 -
Wordpress SEO/Tag plugin recommendation for sports (soccer) website
I own a Wordpress website which covers soccer in the DC MD VA area called DMV Soccer http://www.dmvsoccer.com/ We write weekly recaps where we tag a player who has scored a goal or performed well in a game. For each player, obviously, a tag is created. What I'm looking for is a plugin or solution that would allow me to tag a player, but also automatically assign a team to that player so that the team name and player's name are optimized on the individual player's tag page. So if I were to tag George Murphy on a recap, and I assign him to a team, let's say DC United. The tag page would have a title, something like: George Murphy Soccer Player for DC United and the meta description: George Murphy, soccer player from MD who players for DC United archives Or something similar, if that makes sense. Should I skip using tags and instead start assigning each player as a sub-category under each team? I'd like to try to avoid that, because not each category will be based on a player. Any suggestions in terms of existing plugins or other recommendations?
Web Design | | georgetsn1 -
Domain Authority Drop After Website Relaunch
Prior to my website redesign and relaunch on July 10th, our domain authority was 33. 301 redirects were implemented properly. Out or 600 pages, about 200 URLs were modified. Domain authority has dropped to 28. Rankings are terrible. Conversions are awful. What does the domain authority drop mean? I have noticed in the past that a drop in domain authority tends to coincide with more a drop in ranking and a drop in the quality of visitors. The site is www.metro-manhattan.com Thanks,
Web Design | | Kingalan1
Alan0 -
Need help in website URL Structure
I have been working on a brand new website currently it is live but I have disallow Googlebots temporarily as I dint want any negative impact. The business of the site is to generate leads , they install and sell Stairlifts and used Stairlifts. There are two main categories New Stairlifts and Reconditioned Stairlifts Currently the URL for new Stairlifts is : http://willowstairlifts.co.uk/stairlifts/ and for Reconditioned Stairlifts is: http://willowstairlifts.co.uk/reconditioned-stairlifts/ My concerns are that the word Stairlifts is mentioned twice in the urls so is it going to have a negative impact or panda penalty? I am thinking of changing them to http://willowstairlifts.co.uk/new/ and the product pages to display as http://willowstairlifts.co.uk/new/brooks/ Currently its http://willowstairlifts.co.uk/stairlifts/brooks/ Same with reconditioned Stairlifts I like to change it to : http://willowstairlifts.co.uk/reconditioned Also its product pages to http://willowstairlifts.co.uk/reconditioned/brooks/ As currently its http://willowstairlifts.co.uk/reconditioned-stairlifts/brooks/ Thanks
Web Design | | conversiontactics0 -
Looking for a website review.
Hello, Our web team and I have been building a e-commerce website for the past couple of months. The project is nearing its end and I was just wondering if there are any big issues on this website that jump out at anyone, besides the fact that it hasn't been submitted anywhere or there has been no link building done yet. The website is wheelchairparts.com All feedback and constructive criticism is appreciated! Thanks!
Web Design | | Mike.Bean0 -
Website award competition
Does anyone know of any good quality website award/competition type site? Somewhere where we can submit our site and try to win an award?
Web Design | | waqid1 -
Site-wide footer links or single "website credits" page?
I see that you have already answered this question before back in 2007 (http://www.seomoz.org/qa/view/2163), but wanted to ask your current opinion on the same question: Should I add a site-wide footer link to my client websites pointing to my website, or should I create a "website credits" page on my clients site, add this to the footer and then link from within this page out to my website?
Web Design | | eseyo0