Development site accidentally crawled - Will this cause problems?
-
We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com). There has been no SEO and no links created to this site, or so I thought.
Recently, I found out that Google somehow found its way to this development site and has been indexing the pages! I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain.
I immediately created a robots.txt file to disallow access to the entire development domain. My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain.
We're slated to launch in 2-3 weeks. Is there anything else I should do? Should I even be worried? I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts.
Thank You!
-
Great Answer, thanks Phil! One follow-up question:
In my robots.txt for the development site, I have the following:
User-agent: *
Disallow: /
Is this the correct configuration for the robots.txt file to accomplish what I want, that being removing the entire site from being crawled and from the exiting index? Or should I be configuring it differently?
Also, good tip on Webmaster Tools. I'll be request removal there as well.
-
I don't even worry about that anymore. I let Google see me build out a site anyway. I used to worry about that, but not anymore.
"I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain."
They probably came to the server and hit every site on it.
-
Setting a Robots.txt file for the Dev Site to be No index was a correct response. You can also add a No index no follow meta tag to the Dev site as well.
Another step you can take is to set up a Google Webmaster Tools account for the Dev site and block there as well.
Some dev sites are placed behind a firewall or require a sign on to access, this process can block google as well.
The risks you have is essentially creating an entire duplicate of your current website. Google will always try and crawl everything it can on the net regardless of Noindex tags. No index simply means please dont place in your index. It is important to remember that there are other Search Engines out there besides Google, Bing/yahoo, Ask, Blekko, etc... and all do not automatically honor the Noindex no follow tag. So any secure pages or documents should be just that - secured.
If those pages are no longer in the index, and are not security or confidential in nature I wouldn't worry too much.
- Phil G
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Redesign Just Starting - Should I Leave The Previous Incomplete Site or Setup A Temporary Holding Page and Redirect Previous URL'S?
Hi All I've picked up a new website project and wanted to ask about the best way to proceed with the current site during the development process. The current site is incomplete although it has been live for a while and has over 80 pages in the sitemap. Link to site https://tinyurl.com/ychwftup The business owner wants to take down the current site and simply add a landing page stating "new website coming soon". From an SEO perspective, am I better to keep the current site live until the new site is ready? Or would it not make any difference if I setup the landing page and add 301 redirects from each page in the sitemap to the landing page. Many Thanks In Advance For Any Assistance
Web Design | | ruislip180 -
Looking for a new Web Developer and Graphic Designer - Where to look? Job Description in Post
Hi All, I know I can go to Guru.com, etc. but is there a place here that I can look for a web dev that can help with my tasks? I have a good bit of work to offer them and my current web dev who has been a great asset has suffered from health problems and it has affected his output, so I'm sad to say I need to look for someone that can help me get caught back up. If anyone has any suggestions on someone that is a knowledgeable web developer that has access to their own graphic designer (or can do graphic design themselves as well) please let me know. I am not opposed at all to hire a separate graphic designer to send work over to the web developer. SOMEONE WITH SEO EXPERIENCE WOULD BE VERY HELPFUL AS WELL AS MY CURRENT WEB DEVELOPER HAD IT AS WELL. The current projects going on are: Changing the header graphics on the home page to meet compliance. Constantly adding and editing my websites with small tweaks almost daily. Creating PDF's with content we give and uploading them as well as images to the sites. Keeping in mind that editing the site can affect responsiveness and we have to make sure our logo is a certain size compared to other metrics, etc. Creating web banners, tile ads, etc. Creating facebook main photo for biz page. I will be happy to share any info via PM or if anyone can point me in the right direction in finding someone that would be awesome! Thanks!
Web Design | | Veebs1 -
Responsive Design vs Mobile For This Site?
They are going to do an entire website redesign for http://gaport.com/ and I think they should adhere to responsive design best practices. However, I'm just saying that because everything I have read says that's the "way of the future" if not the way of the present already. Any reason, they shouldn't do that and keep the desktop/mobile sites? Thanks, Ruben
Web Design | | KempRugeLawGroup0 -
What's the point of an EU site?
Buongiorno from 18 degrees C Wetherby UK 🙂 On this site http://www.milwaukeetool.eu/ the client wants to hold on to the EU site despite there being multiple standalone country sittes e.g. http://www.milwaukeetool.fr & http://www.milwaukeetool.co.uk Why would you ever need an EU site? I mean who ever searches for an EU site? If the client holds on to the eu site despite my position it's a waiste of time from a search perspective is the folowing the best appeasment? When a user enters the eu url or redirects to country the detected, eg I'm in Paris I enter www.milwaukeetool.eu it redirects to http://www.milwaukeetool.fr. My felling this would be the most pragmatic thing to do? Any ideas please,
Web Design | | Nightwing
Cioa,
David0 -
Will upgrading my dedicated server improve my site speed
hi, at the moment i am concerned about my site speed for www.in2town.co.uk My hosting company is tmd hosting and my package is Intel Atom 330 1MB L2 Cache 1.6GH $159/mo $189/month500GBStorage4GBRAM10TBBandwidthi am looking at increasing this to**$219/mo** $289/month500GBStorage6GBRAM10TBBandwidthcan anyone let me know if this will make a difference to my site speed please
Web Design | | ClaireH-1848860 -
Site Re-Design - Running old XML site map for 301's
Hi all, We are going to launch a new site design for our current e-commerce site. I have taken this opportunity to change some categories due to keyword research and all old categories will be 301ed to best fitting new category. So I have 2 questions about moving stuff over; 1. I read that leaving the old xml site map running for the first week, would help, because this would give crawlers the chance to run through the site and follow the 301s, which would help pass the juice. How true does this sound? 2. I was thinking of re-writing all category and sub category titles, meta descriptions and on page content. The positive of this is loads of fresh content - but doing this all at the same time with the new site launch might see some major dropping in search ranking. I've identified our top traffic keyword terms/pages, would it be more wise to leave these pages, and change the others, or would the total new fresh burst have a better impact? Cheers
Web Design | | ToxicFox0 -
How will it affect my site if i link to a site with adult content?
We are currently working on creating 2 sites for a company, one with no adult content, one with adult content. Will it affect the non adult content site if i link to the other one in terms of Google and being blocked by some internet providers.
Web Design | | MattWheatcroft0 -
What site do you admire/like for its SEO - technical, content, whatever - and why?
I am gathering examples of great SEO'd sites and would appreciate your examples. The rationale can be anything - great SEO structure, great linking, solid content - you think stands out. Thank you!
Web Design | | josh-riley0