How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What type of website is best for seo.
I need a new website for my health insurance business. What type is best for SEO? Many thanks
Web Design | | laurentjb0 -
Why is google still crawling my old website pages?
Why is google still looking at my old indexed pages and not my new index. ? Why are they crawling my old website links when none of them are available? How do I overcome these problems?
Web Design | | optimalspaces0 -
Blog on Subdomain or Subdirectory?
I was wondering what the SEO impact was of having a blog on a sub domain or sub directory. Which of the following is better? blog.companysite.com companysite.com/blog Personally, I like making the blog a more important part of the site so 2) is preferable; however, I don't know which one is best for SEO purposes. Do the search engines consider a subdomain as a different website? Can SEO authority be passed through each equally? What is your preference? Regards, Dino
Web Design | | Dino640 -
Website Blog causes duplicate pages
Hello, I added a blog to my website, which is hosted at weebly. I was told this would drive traffic but I have actually fallen way, way down in Alexa rankings. When I ran a campaign here, the results show over a 100 errors, all to do with the website blog. It states they are duplicate pages and titles. I dont see a way to rename the pages. Am I better off getting rid of the blog? Thanks
Web Design | | Gardengirl0 -
I need to buy a website template - anyone knows a good place to buy it? where you should buy one template?
I used the www.templatemonster.com or http://themeforest.net/, but is difficult to find a template with a focus on SEO. Also with the necessity to include text in the beginning of the website, is difficulty to found something really good.
Web Design | | Naghirniac0 -
Website using javascript to serve up content - SEO Friendly?
I'm checking out a dentist website http://www.sagedentalnj.com/ I was referred by a friend so just taking a little peek at it. When you click on the menu items, the url at the top doesn't change. When you view source, the page titles are all the same. when I do site:http://www.sagedentalnj.com/ none of his pages are indexed by google. What can be done with his site so that google sees his pages? Maybe submit sitemap?
Web Design | | Czubmeister0 -
Technical SEO Question about TLD combined with SubDomain
I am making a new website but need to figure out the best way to do this in terms of SEO. I would like the website to have functionality of brochure website combined with an online store. My issue is that I will be using software called prestashop for my online store and CMS called MODx to develop my brochure site. (These can not be combined into one CMS). I can create brochure site with MOdx = www.example.com & then from that a subdomain using prestashop for my online store = store.example.com Can I get Google to index these as one site or would I be better off trying to get everything under the TLD. Ideally I would like just one site without subdomain Bacially what I am asking is... What are the effects of having subdomains in terms of SEO? Am I better of having everyhitng under TLD? Can I get Google to view TLD and Sub as one site? Hope this makes sense, thank you.
Web Design | | Socialdude0 -
Could Website redesign be a cause of drop in rankings?
We had a complete redesign of our website and moved it over to wordpress several months ago. As url's changed, we had appropriate 301 redirects done. Rankings for our top keywords dropped, but others remained intact. Our SEO company told us rankings drop when a redesign is done, but I thought if we did all redirects properly (which they approved), it wouldn't be much of a problem. Additionally, we've been steadily adding good new content. Any advice?
Web Design | | rdreich490