How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Blogspot blog is subdomain, but domain is changing - How will this affect backlinks?
Hi Moz community, I appreciate the title is confusing, so let me explain. We use blogspot to host our blog. It's set up as a subdomain of our website. Let's call it: lovelyblog.lovelytraining.com Our website, in this example, is **lovelytraining.com ** We're migrating our website to a new domain: lovely.training We'll be redirecting everything on the primary website, so our link profile won't be lost. However, as the blog is hosted on blogspot, we'll lose links pointed to the blog. The blog would remain lovelyblog.lovelytraining.com - but our website would now be lovely.training The question is, has anyone migrated/redirected a blogspot blog in this way, to retain links? Secondly, is there another way we can tell Google that this is blog should be treated as a subdomain of our website? I'm sure I'm missing out something stupid, so don't go easy on me! Thanks all.
Web Design | | RobertChapman0 -
Should Blog Category Archive URLs be Set to "No-Index" in Wordpress?
It appears that Google Webmaster Tools is listing about 120 blog archives URLs in Google Index>Index Status that should not be listed. Our site map contains 650 pages, but Google shows 860. Pages like: <colgroup><col width="464"></colgroup>
Web Design | | Kingalan1
| http://www.nyc-officespace-leader.com/blog/category/manhattan-office-space | With Titles Like: <colgroup><col width="454"></colgroup>
| Manhattan Office Space Archives - Metro Manhattan Office Space | Are listed when in the Rogerbot crawl report for the site. How can we remove such pages from Google Webmaster Tools, Index Status? Our site map shows about 650 pages, yet Google show these extra pages. We would prefer that they not be indexed. Note that these pages do not appear when we run a site:www.nyc-officespace-leader.com search. The site has suffered a drop in ranking since May and we feel it prudent to keep Google from indexing useless URLs. Before May 650 pages showed on the Webmaster Tools Index status, and suddenly in early June when we upgraded the site the index grew by about 175 pages. I suspect the 120 blog archives URLs may have something to do with it. How can we get them removed? Can we set them to "No-Index", or should the robot text be used to remove them? Or can some type of removal request be made to Google? My developers have been struggling with this issue since early June. The bloat on the site is about 175 URLs not on the site map. Is there any go to authority on this issue (it is apparently rather complicated) that can provide a definitive answer? Thanks!!
Alan0 -
Do you feel my website looks outdated?
My real estate website is www.RiverValleyGroup.com I'm considering undergoing a new design...bounce rate is somewhat high and I can't decide if the current design looks outdated or not. Thoughts?
Web Design | | gohawks77900 -
CMS dynamicly created pages indexed?
Hey Moz'erz, Looking at the indexed pages of my clients eCommerce website I noticed that dynamically created pages are being indexed. For example this page does not "exist" but is created by a drop down filter menu that sorts by product tag: /collections/tools/TAG I can only conclude that this page got indexed either through a backlink or once upon a time there was an internal link pointing to this URL and got indexed (currently there is not). Are either of these cases possibilities? In either case before considering removal or any action I would of-course reference analytics to check for conversions, traffic and any backlinks for those "pages". I believe at the end of the day is recommend a drop down filer that doesn't create new pages as the best solution. Thoughts, comments and experience is greatly welcomed 🙂
Web Design | | paul-bold0 -
Will google penalize a website for using a table layout?
I just got a new client today and his entire website layout and structure is using tables instead of divs. This client is on a tight budget and wants to avoid unnecessary hours for re-coding the website, but at the same time he wants me to improve his SEO organically. This is the first time I've been asked to do work on an existing website that uses pure tables for the entire layout and I'm wondering if this effects the SEO in any way. So my question is, will tables effect rankings and SEO in any way?
Web Design | | ScottMcPherson0 -
Website spread over two domains - The Good,The Bad and the Ugly ?
Hi, I am wondering if the way I have set up our website site is creating more harm than good ? All our SEO efforts and content is located @ www.sparrowmakeup.com.au however our portfolio is located at www.make-up-artists.com.au. Questions ? Will Analytic bounce rates effect my SERP, as most people click on the portfolio link in the first 5-15 sec of landing on our page ?? Is there a better way of setting it up ? the reason I set it up like this in the first place was because we wanted a different look/theme for the portfolio. (using WP) Is there any other issues that this setup might have in the SEO / SERP department ?? Is there any benefits and how can I maximize on them (silver lining) Your help and experience is greatly appreciated. Regards Ed
Web Design | | EdsonGroupMedia0 -
Mobile website strategy
Hello all! This question is about mobile websites. Any suggestions are welcomed. The facts: We have recently built a mobile website for our business using the separate mobile urls (parallel mobile website on a m. sub-domain). We are in the service industry and while our customers are most of the time on the go, it is important for them to find the price and order the service as fast and easy as possible. The redirect from desktop to mobile happens when customers are accessing the website with a mobile devices and it's done automatically. The mobile version does not have any content and serves only as a search and order function. The SEO has been made for the desktop website only and we are ranking really well for all competitive keywords in our niche. We want to keep the mobile website simple and clean so we are not planning to add any content on it.The questions: Since we are not planning to have much content on the mobile website, do we need to do any SEO for it? If yes, why? What are the benefits? Should we add content on the mobile website and why? Is there any down-side from an SEO perspective due to the fact that the website redirects to the mobile version? How are the inbound links going to behave in this case? Do we need to link these two websites although there is no risk about duplicate content? (links, canonical tags, sitemaps) Will the mobile website take advantage of the link building made for the desktop version? Should we allow the mobile website to be indexed having the fact that this is just a search & book tool? (at the moment it is blocked via robots.txt) Thanks for reading, looking forward to your answers.
Web Design | | echo10 -
Best Website Builder - Help Me Choose
I need to built a multi language site (to built a Pilates, Yoga site) and I will use a site builder. After posting questions on wix.com I came to the fact I should continue my research because there are not SEO friendly. Do you have a suggestions? Limited to html knowledge, using a website builder is my only option. Here are some of the features I need: Multilanguage Web Site Mobile version SEO Friendly Nice Template Selections( this is important) HTML customization Twitter, Facebook, Blog... I'm not looking at free website builder, when you want good features, there is a price to paid. Thank you for your help and suggestions, BigBlaze
Web Design | | BigBlaze2050
