Development site accidentally crawled - Will this cause problems?
-
We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com). There has been no SEO and no links created to this site, or so I thought.
Recently, I found out that Google somehow found its way to this development site and has been indexing the pages! I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain.
I immediately created a robots.txt file to disallow access to the entire development domain. My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain.
We're slated to launch in 2-3 weeks. Is there anything else I should do? Should I even be worried? I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts.
Thank You!
-
Great Answer, thanks Phil! One follow-up question:
In my robots.txt for the development site, I have the following:
User-agent: *
Disallow: /
Is this the correct configuration for the robots.txt file to accomplish what I want, that being removing the entire site from being crawled and from the exiting index? Or should I be configuring it differently?
Also, good tip on Webmaster Tools. I'll be request removal there as well.
-
I don't even worry about that anymore. I let Google see me build out a site anyway. I used to worry about that, but not anymore.
"I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain."
They probably came to the server and hit every site on it.
-
Setting a Robots.txt file for the Dev Site to be No index was a correct response. You can also add a No index no follow meta tag to the Dev site as well.
Another step you can take is to set up a Google Webmaster Tools account for the Dev site and block there as well.
Some dev sites are placed behind a firewall or require a sign on to access, this process can block google as well.
The risks you have is essentially creating an entire duplicate of your current website. Google will always try and crawl everything it can on the net regardless of Noindex tags. No index simply means please dont place in your index. It is important to remember that there are other Search Engines out there besides Google, Bing/yahoo, Ask, Blekko, etc... and all do not automatically honor the Noindex no follow tag. So any secure pages or documents should be just that - secured.
If those pages are no longer in the index, and are not security or confidential in nature I wouldn't worry too much.
- Phil G
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why would a developer build all page content in php?
Picked up a new client. Site is built on Wordpress. Previous developer built nearly all page content in their custom theme's PHP files. In other words, the theme's "page.php" file contains virtually all the HTML for each of the site's pages. Each individual page's back-end page editor appears blank, except for some of the page text. No markup, no widgets, no custom fields. And no dedicated, page-specific php files either. Pages are differentiated within page.php using: elseif (is_page("27") Has anyone ever come across this approach before? Why might someone do this?
Web Design | | mphdavidson0 -
Any body can help me to make my web site seo freindly?
any body can help me to make my web site seo freindly? i have not big budget please email me fabric35@hotmail.com
Web Design | | fabric-fabric0 -
301 Redirect all pictures when moving to a new site?
We have 30,000 pictures on our site. Moz will return 404's on some occasionally, but Google seems to ignore those. Should I 301 redirect all those images when we move to a new site lay-out? Appreciate your views!
Web Design | | Discountvc0 -
Can the website pages have the site name like Title of the page | Sitename.com
Hi, Can the website pages have the site name like Title of the page | Sitename.com I have a site with 50K pages and all pages have | Sitename.com mentioned would that be a good practice or bad? Thanks Martin
Web Design | | mtthompsons0 -
Joomla! Site Returning 12000+ Duplicate Content Errors! W Image
(I do award "Good Answer" and "thumbs up" to responses as earned) I have tried to ask this question previously (maybe not correctly). I have a client that I am doing the on and offsite optimization and the MOZ report is kicking back major errors. I have examples below. They all seem to relate directly to rokecwid and ECWID. Is there ANY solution to fix this? Is this hurting the rankings Since I didn't build the site, I am having to tell the website company what to do when I need changes made to code, etc... I am also not very proficient with Joomla! and my web engineer is one of those closet coders (the best kind to have) and doesn't communicate in a way that a "layman" could understand. He pointed out several issues with the HTML but I don't think that is related to this below. Can anyone tell me what to tell the web company that built this site to get rid of these errors? A very small sample of the urls w errors:
Web Design | | Atlanta-SMO
http://www.metroboltmi.com/shop-spareparts?
Itemid=218&option=com_rokecwid&view=ecwid&ecwid_category_id=3560097
1 14 1 http://www.metroboltmi.com/shop-spareparts?
Itemid=218&option=com_rokecwid&view=ecwid&ecwid_category_id=3560098
1 1 0 http://www.metroboltmi.com/shop-spareparts?
Itemid=218&option=com_rokecwid&view=ecwid&ecwid_category_id=3560099
1 14 1 http://www.metroboltmi.com/shop-spareparts?
Itemid=218&option=com_rokecwid&view=ecwid&ecwid_category_id=3560100
1 14 1 SEOMOZErrors_zps3a1ce2a2.png0 -
Avoiding duplicate content with multi-lagusage site
Hi, We have a client in China that is looking to create three versions of the same website, English, Chinese and Korean. They do not want to use a translation plugin like Google translate, preferring to have the pages duplicated. What is the best way to do this bearing in mind that the site needs to be found in all three languages. Would also appreciate if anyone knows of a good hosting company that has English support on the Chinese main land. Thanks Fraser
Web Design | | fraserhannah0 -
Turning my Design Business site into a site to promote SEO
I need advice on retooling my website for my SEO biz. I have shifted my business model from graphic designer who does websites, to "internet marketing consultant who does graphics too". My main website and domain name is over 10 years old, so I've made the decision to keep it, even though it has no keywords in the name. The name works well for the new business, otherwise. The site has a PR3 and I rank well for small business advertising terms, which gets me graphic design business. I intend to keep doing graphic design, but that is a smaller part of my income. I had considered making 3 satellite sites with keyword domain names to cover my offerings of graphic design SEO, website development, and internet marketing. But am leaning against it for several reasons (that all of us SEO's know) but mainly the fact that I cannot keep up with both working for my clients and blogging on multiple sites and link building for multiple sites. So my question is (you knew there was one coming, right?), what is the best approach to building categories of web development, internet marketing, and SEO into my existing graphic design/advertising oriented website? This is slightly embarrassing to ask as an SEO, but given the multiple approaches possible, and knowing the importance of doing it right the first time, it's best to get an consensus perspective on the BEST approach. My main concerns are the navigation system and the links from the homepage into the site. I have too many pages I've identified as essential to link off of the home page and navigation menus? (Website development, social media marketing, link building, keyword research, pay per click, online advertising, graphic design, brochures, catalogs, Logos, Branding, SEO, keyword research etc.) I've always tried for the ratio of one link off of any page for every 100 words of content. Do I create a home page that is of monster proportions? Do I just have the 4 basic areas linking off the home page then create a "landing zone" of 4 folders and create down from that? I am concerned about URL length as I go deeper with that approach. Or, does it make more sense to have a dozen second-level pages, and not link them all off the home page, and build from beneath (and relying on external juice). Next issue is the nav system. It will be huge. Am I best off just keeping it to 4-6, and creating subnavigation on everypage within the site according to section (PITA)? I've read dozens of blog opinions on how much nav systems do or do not hurt link juice. I've always thought footer links were right next to worthless to pass any juice, but given this situation, does it make sense to make a footer link for each major page (about 20)? Thanks for your opinions.
Web Design | | JCDenver0 -
Can i do this? Will Google penalize me?
I have a page for a Criminal Defense Attorney and i set up a list of the type of criminal charges he is certified to deal with. I wanted to use title tags and put the Keyword "Miami Criminal Defense Attorney" & "Miami Traffic Defense Lawyer"... My question is will Google penalize me for plugging the same Key words over and over on the title tag for each ?? CHECK THE IMAGE to see what I'm talking about... thanks guys. x97dl
Web Design | | marig0