Development site accidentally crawled - Will this cause problems?
-
We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com). There has been no SEO and no links created to this site, or so I thought.
Recently, I found out that Google somehow found its way to this development site and has been indexing the pages! I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain.
I immediately created a robots.txt file to disallow access to the entire development domain. My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain.
We're slated to launch in 2-3 weeks. Is there anything else I should do? Should I even be worried? I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts.
Thank You!
-
Great Answer, thanks Phil! One follow-up question:
In my robots.txt for the development site, I have the following:
User-agent: *
Disallow: /
Is this the correct configuration for the robots.txt file to accomplish what I want, that being removing the entire site from being crawled and from the exiting index? Or should I be configuring it differently?
Also, good tip on Webmaster Tools. I'll be request removal there as well.
-
I don't even worry about that anymore. I let Google see me build out a site anyway. I used to worry about that, but not anymore.
"I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain."
They probably came to the server and hit every site on it.
-
Setting a Robots.txt file for the Dev Site to be No index was a correct response. You can also add a No index no follow meta tag to the Dev site as well.
Another step you can take is to set up a Google Webmaster Tools account for the Dev site and block there as well.
Some dev sites are placed behind a firewall or require a sign on to access, this process can block google as well.
The risks you have is essentially creating an entire duplicate of your current website. Google will always try and crawl everything it can on the net regardless of Noindex tags. No index simply means please dont place in your index. It is important to remember that there are other Search Engines out there besides Google, Bing/yahoo, Ask, Blekko, etc... and all do not automatically honor the Noindex no follow tag. So any secure pages or documents should be just that - secured.
If those pages are no longer in the index, and are not security or confidential in nature I wouldn't worry too much.
- Phil G
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Changing top level navigation between site sections
We've got an internal proposal to change our top level nav depending on the section of the site. For example, on our homepage it might read: Products, Library, About with relevant links dropping down below. As we have varied products, the drop down underneath it would include the various families. When arriving on the product family page the top-level nav would change to represent more specific offerings. For example: xxx.com 1. Products; 2. Library; 3. About xxx.com/xxx 1. Product family 1; 2. Product family 2; 3. Product family 3; 4. Library; 5. About What are the SEO/UX implications of this? It seems confusing but allows more specific navigation via the main nav depending on the section of the site. Also it seems that an alternating TLN might not be too Google-friendly.
Web Design | | gwelch0 -
Do Google Fonts Slow Down Your Site?
Hi Guys,
Web Design | | jeeyer
I just did a webpage speed test on http://www.webpagetest.org to see how our site is performing.
I noticed that an exteral URL called fonts.gstatic.com has a "huge" impact on our sites loading time. See a screen here: http://monosnap.com/image/z6drzC2ELoJ48d1rM0Tmtuszl3pFpH#
An overview can be seen here: http://monosnap.com/image/9hofUpr5Ld8D7mi7zyaJmGFIGhpBsY# All our scores are green and A (finally!) but I was a bit concerned when I saw the outcome of the pagespeedtest regarding the fonts.
When I load a page on my pc I indeed notice that the text content is usally quite slow in showing up, pops up afer a few seconds. Is this a know problem and Is this something I need to fix? If so what is the best approach? Looking forward on your thoughts!
Joost1 -
Recovering organic traffic and Google rankings post-site-crash
Hi everyone, we had a client's Wordpress website go down about 2 weeks ago and since then organic traffic has basically plummeted. We haven't identified exactly what caused the crash, but it happened twice in one week. We spent a lot of time optimizing the site for organic SEO, improving load times, improving user experience, improving the website content, improving CTR, etc. Then one morning we get a notification from our uptime monitoring service that the site was down, and upon further inspection we believe it may have been compromised. The child theme that the website was using, all of the files were deleted and/or blank. We reverted the website to a previous backup, which fixed the problem. Then, a few days later, the same exact thing happened, only this time the child theme files were missing after the backup was restored. We've since re-installed and reconfigured the child theme, changed all passwords (Wordpress, FTP, hosting, etc.), and we're looking into changing hosting providers in the very near future. The site uses the Yoast Wordpress SEO plugin, which has recently been reported as having some security flaws. Maybe that was the cause of the problem. Regardless, the primary focus right now is to recover the organic traffic and Google rankings that we've worked so hard to improve over the past few months up until this disaster occurred. The client is in a very competitive niche and market, so I'm pretty frustrated that this has happened after we were making such great progress, Since the website went down, organic search traffic has decreased by 50%. The site and all internal pages are loading properly again (and have been since the second time the website went down), but Google Webmaster Tools is still reporting a number of pages as "not found" witht he crawl dates as early as this past weekend. We've marked all errors as "fixed", and also re-submitted the Sitemaps in Google Webmaster Tools. The website passes the "mobile-friendly" tests, received A and B grades in GTMMetrix (for whatever that's worth), and still has the same original Google Maps rankings as before. The organic traffic, however, and organic rankings on Google have seen a pretty dramatic decrease. Does anyone have any recommendations when it comes to recovering a website's authority and organic traffic after it's experienced some downtime?
Web Design | | georgetsn0 -
Question About Site Redesign and Nav / Page Structure
Hey guys, i am currently redesigning our company's site, and have come across some things that I'm not quite sure of. We used to have individual service pages in our main navigation (design, video, marketing) before the redesign. In this new design, i had the idea of making just one "services" or "capabilities" page, where these three services would each be outlined, and each service would have a list of links to more specific landing pages. Obviously, breaking it up correctly with HTML5 using the andtags. What I'm wondering is that if i'm going to be penalized for having those three services that aren't necessarily related too closely on the same page as opposed to having the one page for each service (like we have now). Any help would be greatly appreciated, and let me know if i need to elaborate more. Thanks in advance!
Web Design | | RenderPerfect0 -
Hey on some of my report cards its saying im not using rel canonical correctly how do i change this on my site?
on some of my report cards its saying certain things featured on my services page are actually linking to my blog or something. and its saying im not using rel canonical correctly. can you help me out?
Web Design | | ClearVisionDesign0 -
Does disabling the "View Source" functionality prevent Google from crawling a website?
I know Google uses a lot of variables when crawling a website. I wasn't sure if disabling the "View Source" option hindered anything.
Web Design | | innovationsimple0 -
Mobile sites! Any advice or suggestions for building one?
Hello Mozland, I'm looking to have a mobile site built. We have a fair amount of traffic coming from smart phones, tablets etc and want to capture this traffic better. Do you have any advice on how to have it built properly, if/how it can be optimised, what good features to include, or anything else you think might be of help? Many thanks Martin
Web Design | | Martin_S0 -
What Is Our Site Missing Causing Our Former Dominance To Slip?
So we have operated one of our retail sites, BonitaJ.com for many years now. Through a lot of work, link building and optimizing around 2009, we were in a prominent spot on the 1st page in google for just about every main term we were targeting. Towards the end of 2009, nearing December or so, we started slipping here and there, and began being displaced for our main terms by newer sites that according to several factors, don't have near the strength our site holds. And by strength, I simply mean, based on link volume, mozbar stats and many other factors, it seems we should rank well above most, but still find ourselves just hanging to 8-10 positions on page one, and in many cases somewhere on page two for terms it seems like we should be in the top 5 positions for. I believe some of our slippage is due to google's devaluing of many of our incoming links. We achieved our early ranking dominence off a lot of directory links and things like that over time, but ever since 2009 when links began getting devalued we immediately broke into getting quality blog links via LEGIT blog relationships where we'd offer up contests, bloggers would review our products and so on, and these relationships continue through today. We also do a lot of guest blog writing, article postings on various networks, as well as press releases, all with the goal of keeping our link profile happy and healthy. So we still have work to do there, but we're on the right track. So my thought is that to get back over the hump, we simply need to continue with the legit link building methods, but I'm also thinking that maybe we need to improve some things navigationally. Things I was hoping people would chime in on are.... 1. If we're mainly trying to target bridal/wedding related jewelry terms, should we ditch the "Jewelry Sets, Pearl Jewelry & Swarovski Crystal Jewerly" terms from our main navbar. They are featured inside each of the categories, and in the end, we don't rank or pull traffic for them anyway. Would ditching them from the main nav, help pass more juice from home page and other pages to the pages that better target our niche? 2. A while back, we ditched including actual product on each of the main category pages. I'm leaning towards breaking the main category pages up into sections, for instance once on the "Bridal Jewelry" page, it would list each of the sub-cats, with a 5-10 product sampling of the most popular items, with a link that says "view all necklaces" at the end of each sub-section. Do you think that more wise than just trying to direct them into the sub-cats with no actual product offering? 3. Anything else you see glaringly wrong with what we're trying to do? This site is just on the edge of blowing up from a ranking perspective if I can just get some confirmation on some things that I know I should do, but I'm wary due to fear of screwing things up. If I can get some solid feedback, the rest is history.
Web Design | | AarcMediaGroup0