How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Directory website creation
Hi, I'm in the process of helping a client with a wedding show & supplier directory. Could anyone give me pointers as to the best practice for the architecture? Because the suppliers and shows will be listed geographically is there a possibility for the site to seem keyword stuffed? thanks, MOZ brain, you guys always rock.
Web Design | | Libra_Photographic0 -
Should I move my blog from subdomain to subdirectory?
Hi there, We have a pretty strong organic presence on our consumer facing blog. Around 5.7 million organic clicks Year to Date. It's currently housed in Wordpress on a subdomain. Our team is considering migrating our blog from Wordpress to Sitecore, where our parent organization resides. With that consideration, we're trying to determine if we should preserve the subdomain or move to a subdirectory of the parent brand. Moving to the parent brand would also result in a loss of our custom global nav - we would inherit the global nav of the parent website. We're not concerned about traffic loss risk - we know that we'll lose some but we think the move to subdirectory might be more beneficial in the long run from an SEO perspective. We're mostly concerned about the users getting lost without the global nav specific to the blog. Thoughts/concerns? Thank you!
Web Design | | Jessdyl0 -
Is the size of website a ranking factor?
I've heard it said that by increasing the number of pages on a website (by adding new content) that this in iself can boost a site's ranking. Is this an old wives tale or is there a grain of truth here? My website - www.skydas.lv metāla durvis Metāla durvju montāža - http://sosdienests.lv
Web Design | | Felter0 -
Why use a wwwP subdomain naming convention
While working through a series of crawl reports and competitive insights for a site, I noticed one of the competitors had switched from a WWW-version to a wwwP-version. Looking back at the snapshot I took of this during the same time period in 2014, I noticed a significant drop in PA/DA by 20+/-. I'm curious to know if anybody else has experienced something similar, and if anybody can provide insights on why a change like this would even be made? I'll preface it with, everything we could see that this competitor was doing from the outside, was legitimate and propelling them in a positive direction.
Web Design | | dodgejd0 -
Lots of Listing Pages with Thin Content on Real Estate Web Site-Best to Set them to No-Index?
Greetings Moz Community: As a commercial real estate broker in Manhattan I run a web site with over 600 pages. Basically the pages are organized in the following categories: 1. Neighborhoods (Example:http://www.nyc-officespace-leader.com/neighborhoods/midtown-manhattan) 25 PAGES Low bounce rate 2. Types of Space (Example:http://www.nyc-officespace-leader.com/commercial-space/loft-space)
Web Design | | Kingalan1
15 PAGES Low bounce rate. 3. Blog (Example:http://www.nyc-officespace-leader.com/blog/how-long-does-leasing-process-take
30 PAGES Medium/high bounce rate 4. Services (Example:http://www.nyc-officespace-leader.com/brokerage-services/relocate-to-new-office-space) High bounce rate
3 PAGES 5. About Us (Example:http://www.nyc-officespace-leader.com/about-us/what-we-do
4 PAGES High bounce rate 6. Listings (Example:http://www.nyc-officespace-leader.com/listings/305-fifth-avenue-office-suite-1340sf)
300 PAGES High bounce rate (65%), thin content 7. Buildings (Example:http://www.nyc-officespace-leader.com/928-broadway
300 PAGES Very high bounce rate (exceeding 75%) Most of the listing pages do not have more than 100 words. My SEO firm is advising me to set them "No-Index, Follow". They believe the thin content could be hurting me. Is this an acceptable strategy? I am concerned that when Google detects 300 pages set to "No-Follow" they could interpret this as the site seeking to hide something and penalize us. Also, the building pages have a low click thru rate. Would it make sense to set them to "No-Follow" as well? Basically, would it increase authority in Google's eyes if we set pages that have thin content and/or low click thru rates to "No-Follow"? Any harm in doing this for about half the pages on the site? I might add that while I don't suffer from any manual penalty volume has gone down substantially in the last month. We upgraded the site in early June and somehow 175 pages were submitted to Google that should not have been indexed. A removal request has been made for those pages. Prior to that we were hit by Panda in April 2012 with search volume dropping from about 7,000 per month to 3,000 per month. Volume had increased back to 4,500 by April this year only to start tanking again. It was down to 3,600 in June. About 30 toxic links were removed in late April and a disavow file was submitted with Google in late April for removal of links from 80 toxic domains. Thanks in advance for your responses!! Alan0 -
Wordpress or custom built website?
Hello fellow mozzers, Hope you are all well. I am looking to get a website done and I am struggling to decide whether to get a custom build website site done or a website built on the WordPress platform. Would you be able to share your experiences and advice/suggestions on what you would use, and why? (pros and cons, etc.) Thanks in advance. Kind Regards
Web Design | | JonathanRolande0 -
Need an SEO pro to build us a new fine art website
Hey Seomoz friends My friends and I started buying and selling art 5 years ago. We did it all on ebay. Two years ago I decided to try an attempt at building us a website to further our business outside of ebay. www.originalartbroker.com I used homestead.com software aka intuit to do this. I was able to get a lot of first page results in seo after a ton of work and willing to do a lot more. I'm sure the homestead site slowed my efforts down. I need more power 🙂 What we do. We sell fine art. We are trying to aspire to the leading seller of fine contemporary art online. We will do whatever it takes to get there. Our issue is not our product. Our issue is getting traffic to the site. I only have 175-200 visitors a day. I need better results for the keywords of our artists and inquires that surround them. We have a bounce rate of 70%, average time on site of 1 minute 50 seconds, and a slow loading site that is almost impossible to create individual pages for pieces (mgmt nightmare) Outside of our site we have blogs, squidoo pages, facebook account with 2500 fans, twitter page with 2k followers, youtube channel, and a blog on the site for new arrivals. We have a 200-250 piece on our site at any given time. We add probably ten pieces on average a week. We don't need any kind of ecommerce management software as most of our sales are done over the phone being that it is higher end art. I need a site that out performs my competition. I need to be in the top three when someone types in "leroy neiman" "peter max" and so on. I, with limited knowledge, need to be able to use the software everyday to upload new art as it comes in. I want every piece to have its own page so that I can also add pieces to a google merchant account. When I add new pieces under a certain artists and it creates its own page i want it to create h tags and an url extension for each product as I add them based off the discription and artists name. I would like some sort of blog integration to post our new arrivals as we get them. I would like some sort of customer capture. I am thinking something along the lines of them prociding email, name, and zip code to see the prices on the site. You are the seo pro so you know what it takes. I would like to know what a solution would cost to get us on the ground with more seo power. A site with speed that is easily indexed. It doesn't need to have a lot of bells and whistles. Please look at my site and let me know what you think. You can get an idea from that of what we are doing. Please give me an idea of what you can do and what it would cost. Thank you
Web Design | | forecastedinvestments0
