How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ranking drop after new website
Hi there, I have a new client who has just had a new website built (by someone else). It was quite a major change as it was 12 years old and has just been moved to Wordpress. However although they are by and large happy with the new site, they have lost a lot of their rankings in Google. The content and menu structure is apparently identical. I told them I didn't think this was unusual but I'm not sure how easy it will be to get them ranking again. Where are they likely to be starting from? Is it a case of starting from the beginning or will there be some residual ranking capability left over? Or can they expect a full recovery over time? I was going to start by looking to see if things like tagging and meta data has been filled in (I will add the site to my Moz account) but is there any way of comparing the old site with the new for SEO purposes? Thanks so much, Sarah.
Web Design | | Frog-Marketing0 -
Advice on estate agent website SEO next steps
Hi everyone, I have been working on the SEO for this website for a while now and have had a good amount of success increasing the traffic and rankings. However our main hurdle is improving conversions on the actual website - we want to encourage more people to book a valuation. Does anybody have any suggestions on how we can improve this? The website is www.richardkendall.co.uk Thank you
Web Design | | sophiecrosby970 -
Traffic Dropping To Website
Hi In Google Analytics:
Web Design | | SEOguy1
I have noticed up to 50% of traffic coming to the website drops off at the home page point,
and drops further from other pages on the site. I realise some may possibly say that this could be down to various factors such as server issues, poor web design, or the wrong traffic reaching the site I have did corrected the following: There was an issue with there being www.domain.com and www.domain.com/home, Screaming Frog and Moz showed that these both had duplicate meta tagging issues. Initially I had created a separate page called 'home' to include in the main nav bar under the slider, but yesterday I replaced this page with a request in the functions.php to place 'home' in the nav bar as a redirect back to the home www.domain.com page. This works great. So I now have the following 301 permanent redirects: non-www to www resolve in the htaccess file, plus 2 permanent 301 redirects in the nav bar. I wonder if this is acceptable protocol re the nav bar redirects, and I wonder if you could possibly advise if the actions that I have taken will have any negative impact on the web seo, link structure, crawlability or indexing. Thanks.0 -
Problems preventing Wordpress attachment pages from being indexed and from being seen as duplicate content.
Hi According to a Moz Crawl, it looks like the Wordpress attachment pages from all image uploads are being indexed and seen as duplicate content..or..is it the Yoast sitemap causing it? I see 2 options in SEO Yoast: Redirect attachment URLs to parent post URL. Media...Meta Robots: noindex, follow I set it to (1) initially which didn't resolve the problem. Then I set it to option (2) so that all images won't be indexed but search engines would still associate those images with their relevant posts and pages. However, I understand what both of these options (1) and (2) mean, but because I chose option 2, will that mean all of the images on the website won't stand a chance of being indexed in search engines and Google Images etc? As far as duplicate content goes, search engines can get confused and there are 2 ways for search engines
Web Design | | SEOguy1
to reach the correct page content destination. But when eg Google makes the wrong choice a portion of traffic drops off (is lost hence errors) which then leaves the searcher frustrated, and this affects the seo and ranking of the site which worsens with time. My goal here is - I would like all of the web images to be indexed by Google, and for all of the image attachment pages to not be indexed at all (Moz shows the image attachment pages as duplicates and the referring site causing this is the sitemap url which Yoast creates) ; that sitemap url has been submitted to the search engines already and I will resubmit once I can resolve the attachment pages issues.. Please can you advise. Thanks.0 -
Subdomains, duplicate content and microsites
I work for a website that generates a high amount of unique, quality content. This website though has had development issues with our web builder and they are going to separate the site into different subdomains upon launch. It's a scholarly site so the subdomains will be like history and science and stuff. Don't ask why aren't we aren't using subdirectories because trust me I wish we could. So we have to use subdomains and I'm wondering a couple questions. Will the duplication of coding, since all subdomains will have the same design and look, heavily penalize us and is there any way around that? Also if we generate a good amount of high quality content on each site could we link all those sites to our other site as a possible benefit for link building? And finally, would footer links, linking all the subdirectories, be a good thing to put in?
Web Design | | mdorville0 -
Mobile website strategy
Hello all! This question is about mobile websites. Any suggestions are welcomed. The facts: We have recently built a mobile website for our business using the separate mobile urls (parallel mobile website on a m. sub-domain). We are in the service industry and while our customers are most of the time on the go, it is important for them to find the price and order the service as fast and easy as possible. The redirect from desktop to mobile happens when customers are accessing the website with a mobile devices and it's done automatically. The mobile version does not have any content and serves only as a search and order function. The SEO has been made for the desktop website only and we are ranking really well for all competitive keywords in our niche. We want to keep the mobile website simple and clean so we are not planning to add any content on it.The questions: Since we are not planning to have much content on the mobile website, do we need to do any SEO for it? If yes, why? What are the benefits? Should we add content on the mobile website and why? Is there any down-side from an SEO perspective due to the fact that the website redirects to the mobile version? How are the inbound links going to behave in this case? Do we need to link these two websites although there is no risk about duplicate content? (links, canonical tags, sitemaps) Will the mobile website take advantage of the link building made for the desktop version? Should we allow the mobile website to be indexed having the fact that this is just a search & book tool? (at the moment it is blocked via robots.txt) Thanks for reading, looking forward to your answers.
Web Design | | echo10 -
Is anyone using Humans.txt in your websites? What do you think?
http://humanstxt.org Anyone using this on their websites and if so have you seen and positive benefits of doing so? Would be good to see some examples of sites using it and potentially how you're using the files. I'm considering adding this to my checklist for launching sites
Web Design | | eseyo1 -
I need to buy a website template with seo focus - any tip? what technology is better?
Due to lack of resources, my company usually buys website templates and we adjust it internally. So we can always be with new designs, spending less money. With all this changes of google panda recently, I will change my website again. Usually we buy then from templatemonster, but some templates are far away from seo standards. Do you have tips where to buy than? Should I search for a specific technology or layout? Is this JS Animated website a goog choice?
Web Design | | Naghirniac0
