Development site accidentally crawled - Will this cause problems?
-
We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com). There has been no SEO and no links created to this site, or so I thought.
Recently, I found out that Google somehow found its way to this development site and has been indexing the pages! I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain.
I immediately created a robots.txt file to disallow access to the entire development domain. My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain.
We're slated to launch in 2-3 weeks. Is there anything else I should do? Should I even be worried? I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts.
Thank You!
-
Great Answer, thanks Phil! One follow-up question:
In my robots.txt for the development site, I have the following:
User-agent: *
Disallow: /
Is this the correct configuration for the robots.txt file to accomplish what I want, that being removing the entire site from being crawled and from the exiting index? Or should I be configuring it differently?
Also, good tip on Webmaster Tools. I'll be request removal there as well.
-
I don't even worry about that anymore. I let Google see me build out a site anyway. I used to worry about that, but not anymore.
"I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain."
They probably came to the server and hit every site on it.
-
Setting a Robots.txt file for the Dev Site to be No index was a correct response. You can also add a No index no follow meta tag to the Dev site as well.
Another step you can take is to set up a Google Webmaster Tools account for the Dev site and block there as well.
Some dev sites are placed behind a firewall or require a sign on to access, this process can block google as well.
The risks you have is essentially creating an entire duplicate of your current website. Google will always try and crawl everything it can on the net regardless of Noindex tags. No index simply means please dont place in your index. It is important to remember that there are other Search Engines out there besides Google, Bing/yahoo, Ask, Blekko, etc... and all do not automatically honor the Noindex no follow tag. So any secure pages or documents should be just that - secured.
If those pages are no longer in the index, and are not security or confidential in nature I wouldn't worry too much.
- Phil G
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New To SEO Management, I just want to double check that my idea will work.
I am new to SEO management. I had a 3 month SEO copy writing internship and a 5 month SEO temp job. In both I mostly wrote copy, but I've been teaching myself SEO on the side, I became Google certified. I ended up getting a telemarketing job and somehow the conversation of SEO came up and I winded up managing their SEO for 12 dollars an hour. They say that every lead generated from the website that turns into a sale will be worth 10 dollars and if and when the sales exceed my paycheck I will starting making commission so long as it stays above my hourly. SEO is very fun and this is like my dream job. They are leaving the planning 100% up to me and I want to make sure that what I am doing will work. My plan is as follows: Part 1: Page Authority via backlinks and social media We are health care brokers and my boss, the owner has a lot of contact. He is talking with large unions like, "The Teamsters," and large company retirment groups like, "Blue flame," which is apparently in some way connected to DTE or GE. Long story short, I am trying to get him to convince them to give us a back link to our main page. He also has a ton of clients that own companies. This is good because they may be persuaded to give us backlinks too. In addition, the tech guy thinks he can implement something where we can get a google +1, facebooks likes/shares, twitter likes and shares and pintrest pin it's that would be a part of an email that we send to people within the list of 12,000 clients. From what I can see, from the client base and the people we are working with we should be able to raise the page authority substantially despite the fact that the site is only a few months old and is not yet out of the sand box. I have been slowly picking off each error with SEO MOZ's website crawling. Part 2: Making a Insurance Jargon Dictionary Guide For The Tri-purpose of gathering traffic, proving our professionalism and helping people understand semi-complex insurance jargon. I could build these 2-3 keywords would be addressed per page and they would be defined in a way to help people looking for terms understand them, while simultaneously netting a strong keyword density and a strong page. I think as far as I can tell there are no issues. Part 3: The dictionary pages will pull in new traffic and the home page will receive links and distribute link juice to the sub-pages. This subpages will guide traffic back to the main page with no-follow links to direct people from the unique termed landing pages to the home page for insurance processing. As far as I can tell my logic is solid and on paper this should work. Am I missing anything (like key details, flaws in my plan)?
Web Design | | Tediscool0 -
ECWID Ecommerce Sites. No Custom URLS?
Is there any way possible to be able to name product urls in website that use ECWID for their ecommerce? They have long and "dirty" urls. For example this running boards site: http://www.runningboards4less.com/general-motors#!/~/product/category=6593890&id=28043027 Isn't this hurting the overall SEO of the site? Especially product pages?
Web Design | | Atlanta-SMO0 -
404's and a drop in Rank - Site maps? Data Highlighter?
I managed an old (2006 design) ticket site that was hosted and run by the same company that handled our point of sale. (Think, really crappy, customer had to click through three pages to get to the tickets, etc.) In Mid February, we migrated that old site to a new, more powerful site, built by a company that handles sites exclusively for ticket brokers. (My site: TheTicketKing. - dot - com) Before migration, I set up 301's for all the pages that we had currently ranked for, and had inbound links pointing to, etc. The CMS allowed me to set every one of those landing pages up with fresh content, so I created unique content for all of them, ran them through the Moz grader before launch, etc. We launched the site in Mid February, and it seemed like Google responded well. All the pages that we had 301's set up for stayed up fairly well in rank, and some even reached higher positions, while some took a few weeks to get back up to where they were before. Google was also giving us an average of 8-10K impressions per day, compared to 3000 per day with the old site. I started to notice a slow drop in impressions in mid April (after two months of love from Google,) and we lost rank on all our non branded pages around 4/23. Our branded terms are still fine, we didn't get a message from Google, and I reached out to the company that manages our site, asking if they had any issues with their other clients. They suggested that I resubmit our sitemaps. I did, and saw everything bump back up (impressions and rank) for just one week. Now we're back in the basement with all the non branded terms once again. I realize that Google could have penalized us without giving us a message, but what got me somewhat optimistic was the fact that resubmitting our sitemaps did bring us back up for around a week. One other thing that I was working on with the site just before the drop was Google's data highlighter. I submitted a set of pages that now come back with errors, after Google seemed to be fine with the data set before I submitted it. So now I'm looking at over 300 data highlighter errors when I'm in WMT. I deleted that set, but I still get the error listings in WMT, as if Google is still trying to understand those pages. Would that have an effect on our rank? Finally I do see that our 404's have risen steadily since the migration, to over 1000 now, and the people who manage the CMS tell me that it would have no effect on rank overall. And we're going to continue to get 404's as the nature of a ticket site would dictate? (Not sure on that, but that's what I was told.) Would anyone care to chime in on these thoughts, or any other clues as to my drop?
Web Design | | Ticket_King0 -
How Can I Make My Site iPhone Friendly?
I have been looking into making my website for iphone friendly as my analytics are not great for the iphone and I know when I try to navigate around it on an iphone it can be tough. I was told that if I make changes to the layout that it would affect my layout across everything, which I did not want to do. So I have two questions: Is this correct regarding the layout? If so, if you did something like m.waikoloavacationrentals.com which would be the mobile version how would that possibly effect your rankings with regards to the traffic distribution? Any feedback would be appreciated. Also if anyone has any experience in doing this I would be interested in discussing further.
Web Design | | RobDalton0 -
Looking for a developer with Network Solutions platform experience
Looking for a developer with Network Solutions platform experience. 714-744-1926Tony Ashford
Web Design | | OCFurniture0 -
Is anyone here managing or doing SEO for a site using GoECart?
We are preparing to update/migrate to a new ecommerce platform. We are in the process of choosing right now. One of the things we know we want is faceted navigation, but I am well aware of the problems this presents for SEO. Are any of you amazing people here using, managing or have experience with GoECart? I am interested to know your feedback, particularly from an SEO viewpoint. Thanks in advance! Dana
Web Design | | danatanseo0 -
Is there something fundamentally wrong with our site architecture?
Hi everyone! Could a few of you brilliant people take a look at the architecture of this site http://www.ccisolutions.com, and let me know if you see any obvious problems? I have run the site through XENU, and all of our most important pages, including categories and products, are no deeper than level 3. Everything deeper than that is, in most cases, an image, a pdf or an orphaned page (of which we have thousands). Could having thousands upon thousands of orphaned pages be having a more hurtful effect on our rankings than our site architecture? I have made loud noises and suggested that duplicate content, site speed and dilution of page authority due to all those orphaned pages are some of the primary reasons we don't rank as well as we could. But, I think those suggestions just aren't sexy or dramatic enough, so there is much shaking of heads and discussion that it must be something fundamentally wrong with site architecture. I know re-arranging the furniture is more fun than scrubbing the floors, but I think our problems are more about fundamental cleanup than moving things around What do you think?
Web Design | | danatanseo0 -
What is a really great bounce rate for a product or service site? What does Good look like?
I am really curious about a result I have never seen before. Our bounce rate went down a lot on a new site. So, what is good??? Recently, we took on a project with a company that offers a product they install for consumers and who had been in business for 15 plus years. The company is successful, has good customer base of those who have been made very happy, etc. It is not a repeat sale type of product, etc. One and done. Their site when we began talking was roughly a year old and was not well constructed but not terrible. Most of the issues were around I frames, use of older coding, poor SEO, etc. There was not really a way to "redesign" and we built a new site. This became a true collaboration in a B2B environment as the owner pushed us like crazy. Not the bad kind of push, the one that makes you say to your team, "Let's find a way!" The result, IMO, was a gorgeous site. But, as you know, those are a dime a dozen. But, to get to the point, when we took over the account they had a bounce rate of around 45%. I did not see this as either good or bad, but a fact and for this industry probably not bad at all. In all honesty, I was not looking at that as a first metric I wanted to move, but it was obviously at or near the top for all the reasons we know. So, this site is a local business, not an everyday product and gets about 2500 to 3000 uniques per month. If we compare to May of 2011/2012: 2011 2012 Total Visitors 1852 3,298 Uniques 1609 2,740 Pageviews 5,634 23,203 Pages/visit 3.04 7.04 Avg Duration 2.05 3.20 Yes, I am leaving off what we are getting, yes, I am leaving off the site. Please don't hate me. I am really wanting to see what others see with site changes and bounce rates first and will disclose. So, what's a great bounce rate? How do you know?
Web Design | | RobertFisher0