Development site accidentally crawled - Will this cause problems?
-
We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com). There has been no SEO and no links created to this site, or so I thought.
Recently, I found out that Google somehow found its way to this development site and has been indexing the pages! I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain.
I immediately created a robots.txt file to disallow access to the entire development domain. My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain.
We're slated to launch in 2-3 weeks. Is there anything else I should do? Should I even be worried? I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts.
Thank You!
-
Great Answer, thanks Phil! One follow-up question:
In my robots.txt for the development site, I have the following:
User-agent: *
Disallow: /
Is this the correct configuration for the robots.txt file to accomplish what I want, that being removing the entire site from being crawled and from the exiting index? Or should I be configuring it differently?
Also, good tip on Webmaster Tools. I'll be request removal there as well.
-
I don't even worry about that anymore. I let Google see me build out a site anyway. I used to worry about that, but not anymore.
"I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain."
They probably came to the server and hit every site on it.
-
Setting a Robots.txt file for the Dev Site to be No index was a correct response. You can also add a No index no follow meta tag to the Dev site as well.
Another step you can take is to set up a Google Webmaster Tools account for the Dev site and block there as well.
Some dev sites are placed behind a firewall or require a sign on to access, this process can block google as well.
The risks you have is essentially creating an entire duplicate of your current website. Google will always try and crawl everything it can on the net regardless of Noindex tags. No index simply means please dont place in your index. It is important to remember that there are other Search Engines out there besides Google, Bing/yahoo, Ask, Blekko, etc... and all do not automatically honor the Noindex no follow tag. So any secure pages or documents should be just that - secured.
If those pages are no longer in the index, and are not security or confidential in nature I wouldn't worry too much.
- Phil G
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Looking for a new Web Developer and Graphic Designer - Where to look? Job Description in Post
Hi All, I know I can go to Guru.com, etc. but is there a place here that I can look for a web dev that can help with my tasks? I have a good bit of work to offer them and my current web dev who has been a great asset has suffered from health problems and it has affected his output, so I'm sad to say I need to look for someone that can help me get caught back up. If anyone has any suggestions on someone that is a knowledgeable web developer that has access to their own graphic designer (or can do graphic design themselves as well) please let me know. I am not opposed at all to hire a separate graphic designer to send work over to the web developer. SOMEONE WITH SEO EXPERIENCE WOULD BE VERY HELPFUL AS WELL AS MY CURRENT WEB DEVELOPER HAD IT AS WELL. The current projects going on are: Changing the header graphics on the home page to meet compliance. Constantly adding and editing my websites with small tweaks almost daily. Creating PDF's with content we give and uploading them as well as images to the sites. Keeping in mind that editing the site can affect responsiveness and we have to make sure our logo is a certain size compared to other metrics, etc. Creating web banners, tile ads, etc. Creating facebook main photo for biz page. I will be happy to share any info via PM or if anyone can point me in the right direction in finding someone that would be awesome! Thanks!
Web Design | | Veebs1 -
How does adding ecommerce to a site affect SEO? What are the negative and what are the positives?
We are thinking of adding ecommerce to our website as a service to our customers. We generate most of our leads through online quote requests but heard that it may be beneficial to our SEO if we add ecommerce for a few products. Is this true? Does anyone have tips on best and worst SEO ecommerce practices?
Web Design | | TeguarMarketing0 -
Hi, I have a doubt. If we want to hide unwanted text in a web page its possible with "" tag. And my question "does a search engine crawl those text? help me.
I want to hide a lot of text behind my site page. I know its possible with that tag. But in what way a search engine looks at those text? Hidden or they are crawled and indexed.
Web Design | | FhyzicsBCPL0 -
Web Developer Using Stock Photos
Hello, The organization is selling a cms system in a niche market across the country. It has the normal SEO challenges, in addition he is using purchased stock images. This seemed ok, while he was smaller but now we are growing rapidly and these images are VERY STOCK- and well used ( I have checked with Tiny Eye). I remember a few years ago this was a flag to the search engines who went through manual review, is this still true? It seems to me that the theme's that come with the images, are duplicated ( including navigation & footers), so having the duplicated images would be another negative. Thank you for your suggestions!
Web Design | | TammyWood0 -
How to determine if my site map needs work?
I recently spoke to a consultant at a search conference who took a look at my site map and mentioned it looked like google would have a hard time crawling the site and indexing new pages and changes. I am managing an ecommerce site with a bunch of products, however, I am not an XML expert by any means so i'd appreciate any advice on what to look for in the site map that would possibly be affecting googles ability to crawl/index.
Web Design | | GregWhiteStarMedia0 -
What is wrong with our site?
Hello Seomoz friends. I've about to pull all my hair out and need to turn somewhere. Our site, www.sightmax.com has been around since 2004. We used to be ranked at the top of page two on google under the keyword "live chat". We are no on page 4, heading to page 5. Can anyone take a look at the site and see if anything jumps out at you? The only way we have been able to get on the first page, is the pay for google adwords (which we've been doing every month for 7 years). Again, the site www.sightmax.com and the keyword is "live chat". Any help or feedback would be appreciated SO MUCH! Thanks! Eric
Web Design | | EricWeber0 -
Could Website redesign be a cause of drop in rankings?
We had a complete redesign of our website and moved it over to wordpress several months ago. As url's changed, we had appropriate 301 redirects done. Rankings for our top keywords dropped, but others remained intact. Our SEO company told us rankings drop when a redesign is done, but I thought if we did all redirects properly (which they approved), it wouldn't be much of a problem. Additionally, we've been steadily adding good new content. Any advice?
Web Design | | rdreich490 -
Recommendation for a company to make a site more mobile friendly?
Hiya, We have a client who uses us for SEO, and a separate company for web development. They have a fairly large site on a bespoke CMS. They're happy with the site, but the user experience on mobile devices is not right. Can anybody recommend a company specializing in that area? Preferably a UK company but not essential. Thanks 🙂
Web Design | | SteveOllington1