How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ranking of non-homepage leads to decrease in website ranking?
Hi all, Google picks up a non-homepage to rank for primary keyword where homepage is actually optimised to rank for same keyword. This means Google is ignoring the actual page and ranking other page. Does this scenario means that we are ranking lower as the homepage is not considered here? We may rank much better if homepage is preferred by Google? Thanks
Web Design | | vtmoz0 -
Website Traffic Drop After Switch
Hey guys, two weeks ago we switched over to the new website, however we've experienced about a 20% drop in Google Organic traffic and it does not appear we are getting indexed correctly by Google. On search results it's not using the meta description and the links still point to the old pages. Robots is fine We are using the same IP address Re-directs are in for the pages in question. Sitemap was submitted to Google in Webmaster tools What else do we need to do?
Web Design | | ScottOlson0 -
Does a Website Overlay Hurt Ranking?
We are looking into personalizing visits on our website. Rather than to write Tb of code on each page, we want to use a little line of Java script on the page, that would allow to place overlay on the site, depending on the visitor. Example: Change the home page picture when a returning visitor comes back on the website and show him/her a different picture bases upon the pages he/she visited prior. Would that overlay be caught by Google and would G have a problem with that? We all know about cloaking, but G does not penalize for heat mapping overlays, for example. Or not to my knowledge they don't. Thanks all for your personal insights! Peter
Web Design | | Discountvc0 -
Is Fall In Keyword Ranking After Launch of Revamped Website Normal
After launching my redesigned website (www.nyc-officespace-leader.com) Google ranking has dropped significantly for competitive keywords. The previous version of the site and the new version both have approximately 450 pages. My website developer was careful to implement 301 redirects. Monitoring Google Webmaster tools it shows that Google has picked up a quantity of duplicate content. More than 950 pages or shown in their index while my site only has 450 pages. There are also certain pages which require canonical which tags my developer is in the process of implementing. The relaunch was July 10. My developer is of the opinion that this fluctuation in ranking is normal and that it will take Google about one month to reindex the new site and remove the old pages from the directory. Is this accurate? Anyone have any ideas on why my site has tanked in Google's search results? Thank you very much. Sincerely,
Web Design | | Kingalan1
Alan Rosinsky0 -
B2C directory website adding B2B ecommerce sub-domain
Hey fellow Mozzers, Just got back from Mozcon and enjoyed getting to know a handful of you. I do in house SEO for a B2B wholesaler. We have a B2C website directory for homeowners to locate contractors to work on their home. On the site we have a products section which includes tech specs but not pricing. Our contractors have been asking us to add the ability to purchase their items online, so we are wanting to add a B2B sub-domain (store.domain.com) to our website for the contractors to purchase products online. We do not want consumers to be able to purchase the items and will have pricing behind a log in. I have a few questions that I'm hoping you might be able to answer: 1. What would be the best practice to not have duplicate content errors with products that are listed on both sites? Should we rel-canonical items shown on both domains or do something else?
Web Design | | AC_Pro
2. We are not against having the new site be crawled, but will Google be upset/ding rankings because pricing is behind a log-in? Are there certain best-practices for B2B ecommerce sites?
3. Do you know of any other sites that have done this/do you have any recommendations on how to best implement this?0 -
Need help in website URL Structure
I have been working on a brand new website currently it is live but I have disallow Googlebots temporarily as I dint want any negative impact. The business of the site is to generate leads , they install and sell Stairlifts and used Stairlifts. There are two main categories New Stairlifts and Reconditioned Stairlifts Currently the URL for new Stairlifts is : http://willowstairlifts.co.uk/stairlifts/ and for Reconditioned Stairlifts is: http://willowstairlifts.co.uk/reconditioned-stairlifts/ My concerns are that the word Stairlifts is mentioned twice in the urls so is it going to have a negative impact or panda penalty? I am thinking of changing them to http://willowstairlifts.co.uk/new/ and the product pages to display as http://willowstairlifts.co.uk/new/brooks/ Currently its http://willowstairlifts.co.uk/stairlifts/brooks/ Same with reconditioned Stairlifts I like to change it to : http://willowstairlifts.co.uk/reconditioned Also its product pages to http://willowstairlifts.co.uk/reconditioned/brooks/ As currently its http://willowstairlifts.co.uk/reconditioned-stairlifts/brooks/ Thanks
Web Design | | conversiontactics0 -
Any advice for best practice for mobile versions of websites?
Hi Mozzers, I am looking to develop a mobile version of one of my sites and was wondering what was considered best practice for the following: Where to host it: m.domain.com or domain.com/mobile ? Is rel='canonical' enough to avoid mobile version being indexed or should I use "noindex" on mobile version? How to handle tablets? Which screen size sees mobile version? Which sees full site version? Thanks in advance Geoff
Web Design | | SEM-Freak1 -
Need to rebuild client's flash website
I am working with their web designer and need to figure out a way to rebuild their site which is currently all in flash. I was wondering if there was a way to do this without spending a ton of time in completely re-doing the site from scratch.
Web Design | | awalker840