Development site accidentally crawled - Will this cause problems?
-
We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com). There has been no SEO and no links created to this site, or so I thought.
Recently, I found out that Google somehow found its way to this development site and has been indexing the pages! I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain.
I immediately created a robots.txt file to disallow access to the entire development domain. My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain.
We're slated to launch in 2-3 weeks. Is there anything else I should do? Should I even be worried? I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts.
Thank You!
-
Great Answer, thanks Phil! One follow-up question:
In my robots.txt for the development site, I have the following:
User-agent: *
Disallow: /
Is this the correct configuration for the robots.txt file to accomplish what I want, that being removing the entire site from being crawled and from the exiting index? Or should I be configuring it differently?
Also, good tip on Webmaster Tools. I'll be request removal there as well.
-
I don't even worry about that anymore. I let Google see me build out a site anyway. I used to worry about that, but not anymore.
"I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain."
They probably came to the server and hit every site on it.
-
Setting a Robots.txt file for the Dev Site to be No index was a correct response. You can also add a No index no follow meta tag to the Dev site as well.
Another step you can take is to set up a Google Webmaster Tools account for the Dev site and block there as well.
Some dev sites are placed behind a firewall or require a sign on to access, this process can block google as well.
The risks you have is essentially creating an entire duplicate of your current website. Google will always try and crawl everything it can on the net regardless of Noindex tags. No index simply means please dont place in your index. It is important to remember that there are other Search Engines out there besides Google, Bing/yahoo, Ask, Blekko, etc... and all do not automatically honor the Noindex no follow tag. So any secure pages or documents should be just that - secured.
If those pages are no longer in the index, and are not security or confidential in nature I wouldn't worry too much.
- Phil G
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I want to create a 301 redirect on a WordPress site, nothing's working...
Hello all, I'm hoping someone out there can give me a hand with this. I'm trying to modify my .htaccess file so that the site will go from maxcarnagemusic.com to www.maxcarnagemusic.com and also, so viewers will be redirected to www.maxcarnagemusic.com/home when they try to access the site. I've tried a few different things, including adding the 301 redirect plugin for Wordpress, but nothing seems to work. Can someone out there show/tell me how to create an htaccess file that will execute as much. I apologize in advance, my Apache experience is very, very limited. Thank you all in advance!
Web Design | | maxcarnage0 -
Thoughts on our Agency Site
Hi all, We'd all welcome opinions on our digital agency site http://www.newbrandvision.com/. We are in the planning stage of launching a new site; and we'd welcome any UX or SEO thoughts. It's a strange one but our agency has been around since 2002, and we operate in the heart of London; however we don't rank anywhere near the first page for "digital agency in London" or any long-tail /semantics around this. We feel that it's pretty clear from search and when landing on the site what we are; but would welcome any general thoughts as to why we aren't ranked that highly. Much appreciated!
Web Design | | Tangent0 -
Moving the site and Rebranding
I was wondering about moving the site and rebranding. If one was to move their site with a good Google Page Rank, how long should you take before doing the updated redesign, cms update and url restructuring? I know that Matt Cutts has said that you should move BEFORE doing your redesign but I don't remember him saying how long you should take for each step. Thanks!
Web Design | | Therealmattyd0 -
Ecommerce Site - SEO
We have a Business Catalyst Site with the Same product Listed in 2 different catalogs. Each product page is the same page with different URLs you can see it here: http://www.yourpharmacy.co.nz/beauty/clarins-skincare/clarins-advanced-extra-firming-eye-contour-cream-20ml http://www.yourpharmacy.co.nz/clarins/clarins-advanced-extra-firming-eye-contour-cream-20ml Any suggestions welcome
Web Design | | OnlineAssetPartners0 -
Changing Links that Show Up when I Google Brand (Site) Name
Hi SEOmoz Community, A quick question for you all. I've added an attachment for reference. When I google my brand name, say for example, Applied StemCell, I see six links as well below the description. Oddly though, these links seem to be chosen at random, or at least I'm not sure how Google decides on them. When I click on one of the links that is the company's name, Applied StemCell it brings me to a PDF document! Is there any way I can choose which ones to display there? Thanks! OF2oVVN.png
Web Design | | swzhai0 -
So apparently SEO moz will get us de-indexed according to a SEO company!
Each and every day i get called up from an SEO company who promises to get me top spots in Google rankings if i quickly get on their special offer they have today normally i would say "no thanks and put the phone down" but i had a bit of spare time so i indulged the guy and we got talking. After the introductions and speal about his company he was showing me what his company does and how they go about it to get me top ranks (they don't get me ranks but create a website they own which then passes leads to me- kinda clever since they could then start charging me per lead or my competitors) We continued to talk and i mentioned i used SEOmoz to check my rankings and back links etc and he told me that Google are cracking down and anyone using these types of software/websites will get their websites de indexed. This struck me as BS but i wanted to get your thoughts on the matter, i personally don't believe Google would ever do such a thing as this since it would be so easy to get your competitors websites taken down (i.e. negative seo) but its certainly a talking point.
Web Design | | GarethEJones0 -
Avoiding duplicate content with multi-lagusage site
Hi, We have a client in China that is looking to create three versions of the same website, English, Chinese and Korean. They do not want to use a translation plugin like Google translate, preferring to have the pages duplicated. What is the best way to do this bearing in mind that the site needs to be found in all three languages. Would also appreciate if anyone knows of a good hosting company that has English support on the Chinese main land. Thanks Fraser
Web Design | | fraserhannah0 -
Site down for more than a month - lost rankings
Hello, We have run into a situtation where we had multiple pages setup for different keywords but didn't realize that we had a name server issue that has caused the pages to be down for the last month or so (2-3 weeks on the low side.) The rank finder was still working fine, but the offline page was never reported. We realized the situation recently and have since gotten the sites back online under the new nameservers. Most of these sites were ranking 1 and 2 spots in their keywords, and now are no where to be found in the Google Index. Should I do anything differently, or just put the sites back online and wait it out? I have seen in different places that it may only take 2 weeks to come back, but it's possible that Google has marked the sites as 'not quality' because of their downtime and it will be even harder to get them to rank again. Can anyone shed any light on this situation? Any information is appreciated. Thanks in advance.
Web Design | | EQ-Richie0