Should I let Google crawl my production server if the site is still under development?

[DoItHappy]

I am building out a brand new site. It's built on Wordpress so I've been tinkering with the themes and plug-ins on the production server. To my surprise, less than a week after installing Wordpress, I have pages in the index.

I've seen advice in this forum about blocking search bots from dev servers to prevent duplicate content, but this is my production server so it seems like a bad idea.

Any advice on the best way to proceed? Block or no block? Or something else? (I know how to block, so I'm not looking for instructions).

• We're around 3 months from officially launching (possibly less).
• We'll start to have real content on the site some time in June, even though we aren't planning to launch.
• We should have a development environment ready in the next couple of weeks.

Thanks!

[DoItHappy]

Thank you for the detailed response, Paul. I'll get cracking on your suggestions.

I was mostly worried that if I blocked it now, it would be mad at me later. You've given me a way to deal with the bot concerns.

I am less concerned that anyone will find these pages. I only knew about their index status because of one of my monitoring services which alerted me that google was crawling.

[ThompsonPaul]

Thanks for the confirmation, Dan! Looks like you're up & working early on a Sunday morning

[ThompsonPaul]

In  my opinion, no, you definitely should NOT allow the production server to be indexed while it's in this state. For all intents and purposes it IS your dev server at the moment, and the last thing you want is for the search crawlers to think that what's there will be representative of the quality of your site when it's finished.

My recommendation:

• get the current site out of the SERPs. (Use WordPress setting in Settings -> Read to check the "Discourage from indexing" box. DON'T add a no-index in robots.txt until the pages have all dropped out of the SERPs)
• when the dev site goes into operation, make _certain_right from the start it cannot be crawled (vastly better than trying to fix the problem after it get's accidentally indexed).
• as soon as you have time, build a proper front page and a few content pages on the production site that indicate what the full site will be about, and get some strong basic, well-written content on there that will also remain after the go-live. (keep ALL the rest of the pages of the prod site out of the SERPs with meta no-index tags)
• once you have a the new, stable, basic content up on prod, allow the SEs to start indexing it.

This gets the messy stuff out of the SERPs before it can pollute the index (and gives you a bad reputation with any actual visitors to the site who shouldn't be seeing your tinkering). By getting some real content as soon as possible, even on a very basic template, you'll start giving the SEs a quality idea of what is to come. Wouldn't hurt to start building a few backlinks once the basic content is up on prod - e.g. links from its new social profiles etc.

This way, when the full site goes live, you'll already have some quality visibility in the engines, so it will be quicker to get the rest of the new site crawled and indexed.

Does that make sense?

Paul

P.S. If at all appropriate, use the basic prod content to show why/how they should connect with you on social media, and offer them a chance to sign up for your newsletter notification of when the site goes live. (It's never too early to start trying to get those subscribers!)