How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Any risks involved when we have huge list of redirects in our website database?
Hi all, We have changed hundreds and thousands of page and page URLs for last 10 years in our website. So, now we are going to redirect most of the old pages/links to the current related pages. This will increase the number of redirects we set in our website database to thousands. Will this put any extra weight on crawler and any risks involved for employing too many redirects. Thanks
Web Design | | vtmoz0 -
Website Form Doesn't Work for Visitors from Google Search App
I've got a strange issue where if visitor is viewing a website from within the Google Search App, the app doesn't seem to be able to handle online forms. The form relies users on entering data, which is then POSTed to an external booking engine site. Normally this works fine, except when Google Search App is acting as the browser, the post payload is empty and the URL breaks. We're a bit stumped as to how to move forward. So far the only lead is that Android users can override having the app behave as a browser - but doesn't seem like iOS users can. Any additional ideas/tips are welcome here - thanks. 🙂
Web Design | | mirabile0 -
Is it important to keep your website home index page simple to rank better?
My website http://www.endeavourcottage.co.uk/ markets holiday cottages and it's grown from my own singular cottage into a small letting agency and I used to rank at best number 3 for the short tailed keywords like Whitby holiday cottages with its drop-down to position 10 on Google.co.uk. So this week I was looking for a UK business to help me improve my rankings and the first thing they said was my home page is detrimental with the listing too many conflicting info with it advertising all 12 properties on it. They suggested a door entry page into the site keeping it simple but when I run it through the analysing tool here on Moz for "Whitby holiday cottages" as an example it came out looking okay. I do the usual things of title tags and meta descriptions for my keywords etc any suggestions or advice would be very welcome thank you Alan
Web Design | | WhitbyHolidayCottages0 -
Website Redesign - Will it hurt SERP?
Hi - I am planning to redesign my blog and I was wondering if this will affect my rankings? The new website template (custom designed) is much more user and seo friendly. The content, url structure, internal linking structure, meta tags, and site structure will remain exactly the same, but the visual design will be different (new sidebar widgets, and slightly different layout on inner pages). The current website is ranking very well (mostly top 5), has a healthy backlink profile, strong social media presence, and great traffic. I have heard that switching to a new template will dramatically hurt the rankings. Is this true? Are there any exceptions? Any ways I can prevent the rankings from dropping? Would really appreciate your input. Thanks in advance. Howard
Web Design | | howardd0 -
AJAX & JQuery Tabs: Indexation & Navigation
Hi I've two questions about indexing Tabs. 1. Let's say I have tabs, or an accordion that is triggered with Jquery. That means that all HTML is accessible and indexed by search engines. But let's say a search query is relevant to the content in Tab#3, while Tab#1 is the one that's open by default. Is there any way that Tab#3 would be open directly if it's more relevant to the search query? 2. AJAX Tabs: We have pages that have Tabs triggered by AJAX (example: http://www.swisscom.ch/en/residential/help/loesung/entfernen-sie-sim-lock.html). I'm wondering about the current best practice. Google recommends HTML Snapshots. A newer SEOMoz Article talks about pushState(). What's the way to go here? Or in other words: How to get Tabs & Accordion content indexed and allow users to navigate directly to it?
Web Design | | zeepartner0 -
Best way to handle Spanish/English WEBSITE
Hey guys, How are you doing? I have a website (www.aceromart.com ) in which the primary language is Spanish. The company is oriented toward the Mexican Audience. However, recently we are dealing with many U.S companies. Also, we want to be included in the Yahoo Directory and several other directories which demand a U.S version of the website. So i want to have a U.S version of the website. My ecommerce is based in the NETWORK Solutions platform and has around 1,000 products and pages. What is the best way to include an English version of the website. Ive seen some pages using the Google traslate tool, which only traslates the text. Hope to hear some of your ideas, Regards,
Web Design | | JesusD0 -
What are some of the best website hosting platforms for Wordpress?
I'm looking for a new hosting provider and have been told to find one that specializes in Wordpress hosting because of higher page speed load times, etc.. Can anyone recommend a couple of Wordpress Hosting providers
Web Design | | webestate0 -
Websites with hidden source data?
Hi, I've noticed that when looking at this site I cannot view their souce data: http://www.lloydspharmacy.com/ All that appears is this: | | |
Web Design | | rossmawd
| | |
| | |
| | |
| |
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | Am I just doing something wrong here? Is this an alternate way of setting up a website and how does it work with regards to SEO? Thanks, Ross0