Should I let Google crawl my production server if the site is still under development?
-
I am building out a brand new site. It's built on Wordpress so I've been tinkering with the themes and plug-ins on the production server. To my surprise, less than a week after installing Wordpress, I have pages in the index.
I've seen advice in this forum about blocking search bots from dev servers to prevent duplicate content, but this is my production server so it seems like a bad idea.
Any advice on the best way to proceed? Block or no block? Or something else? (I know how to block, so I'm not looking for instructions).
- We're around 3 months from officially launching (possibly less).
- We'll start to have real content on the site some time in June, even though we aren't planning to launch.
- We should have a development environment ready in the next couple of weeks.
Thanks!
-
Thank you for the detailed response, Paul. I'll get cracking on your suggestions.
I was mostly worried that if I blocked it now, it would be mad at me later. You've given me a way to deal with the bot concerns.
I am less concerned that anyone will find these pages. I only knew about their index status because of one of my monitoring services which alerted me that google was crawling.
-
Thanks for the confirmation, Dan! Looks like you're up & working early on a Sunday morning
-
In my opinion, no, you definitely should NOT allow the production server to be indexed while it's in this state. For all intents and purposes it IS your dev server at the moment, and the last thing you want is for the search crawlers to think that what's there will be representative of the quality of your site when it's finished.
My recommendation:
- get the current site out of the SERPs. (Use WordPress setting in Settings -> Read to check the "Discourage from indexing" box. DON'T add a no-index in robots.txt until the pages have all dropped out of the SERPs)
- when the dev site goes into operation, make _certain_right from the start it cannot be crawled (vastly better than trying to fix the problem after it get's accidentally indexed).
- as soon as you have time, build a proper front page and a few content pages on the production site that indicate what the full site will be about, and get some strong basic, well-written content on there that will also remain after the go-live. (keep ALL the rest of the pages of the prod site out of the SERPs with meta no-index tags)
- once you have a the new, stable, basic content up on prod, allow the SEs to start indexing it.
This gets the messy stuff out of the SERPs before it can pollute the index (and gives you a bad reputation with any actual visitors to the site who shouldn't be seeing your tinkering). By getting some real content as soon as possible, even on a very basic template, you'll start giving the SEs a quality idea of what is to come. Wouldn't hurt to start building a few backlinks once the basic content is up on prod - e.g. links from its new social profiles etc.
This way, when the full site goes live, you'll already have some quality visibility in the engines, so it will be quicker to get the rest of the new site crawled and indexed.
Does that make sense?
Paul
P.S. If at all appropriate, use the basic prod content to show why/how they should connect with you on social media, and offer them a chance to sign up for your newsletter notification of when the site goes live. (It's never too early to start trying to get those subscribers!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Strange site link on Google for a Facebook result
A Facebook page targetted to US Hispanics (with content in Spanish and English) is showing me a hindi sitelink underneath the main Facebook link when I google (in the US, English) for the page [ page name facebook]. We don't have any content in hindi, or targetted to that audience. If I click on the sitelink while logged out of facebook, I can see it takes me to a facebook subdomain of hi-in. When I'm logged in it just redirects me to the same page. Any idea why this could be happening?
Intermediate & Advanced SEO | | M_80 -
Is possible to submit a XML sitemap to Google without using Google Search Console?
We have a client that will not grant us access to their Google Search Console (don't ask us why). Is there anyway possible to submit a XML sitemap to Google without using GSC? Thanks
Intermediate & Advanced SEO | | RosemaryB0 -
How much SEO damage would it do having a subdomain site rather directory site?
Hi all! With a coleague we were arguing about what is better: Having a subdomain or a directory.
Intermediate & Advanced SEO | | Gaston Riera
Let me explain some more, this is about the cases: Having a multi-language site: Where en.domain.com or es.domain.com rather than domain.com/en/ or domain.com/es/ Having a Mobile and desktop version: m.domain.com or domain.com rather than domain.com/m or just domain.com. Having multiple location websites, you might figure. The dicussion started with me saying: Its better to have a directory site.
And my coleague said: Its better to have a subdomain site. Some of the reasons that he said is that big companies (such as wordpress) are doing that. And that's better for the business.
My reasons are fully based on this post from Rand Fishkin: Subdomains vs. Subfolders, Rel Canonical vs. 301, and How to Structure Links for SEO - Whiteboard Friday So, what does the community have to say about this?
Who should win this argue? GR.0 -
Should you give all the posts in a Forum an unique description? Or let it empty so Google can make one with the crawled keywords .... ...
To make all descriptions for all forum posts unique is a hell of a job.... One option is to crawl the first 165 characters and turn these automaticly into the meta description of the page.
Intermediate & Advanced SEO | | Zanox
If Google thinks the meta description is not suitable for the search query, Google will make a own description. In this case all te meta descriptions are unique, like the Google Guidlines want you to do. How will Google think off the fact when we delete the meta description tag so Google will make all the descriptions by herself?0 -
When crawls occur - when will my links show up in Open Site Explorer
Hello everyone, I've been building links for a while now and none of them show up in Explorer. My domain authority hasn't changed for about a month or so. When does Google do crawls and when does SEOMoz do crawls? Thanks
Intermediate & Advanced SEO | | Harbor_Compliance0 -
Google Site Extended Listing Not Indexed
I am trying to get the new Site map to be picked up by Google for the extended listing as its pulling from the old links and returning 404 errors. How can I get the site listing indexed quickly and have the extended listing get updated to point to the right places. This is the site - http://epaperflip.com/Default.aspx This is the search with the extended listing and some 404's - Broad Match search for "epaperflip"
Intermediate & Advanced SEO | | Intergen0 -
Unsure where Google has sourced this inaccurate Product Data
Hi, This is a slightly odd one I was hoping someone could shed some light on. One of our staff just did a Google search and located these listings on Google UK Product Search: http://www.google.co.uk/search?q=ink+cartridges&hl=en&sa=X&biw=1074&bih=499&tbm=shop&prmd=imvns#q=ink+cartridges&hl=en&sa=X&tbs=store:3287803270081455254&tbm=shop&prmd=imvns&ei=xp5pUP6uN8i_0QXUuoHADQ&ved=0CI0BEMcMMAE&bav=on.2,or.r_gc.r_pw.r_qf.&fp=333b49ec245f6031&biw=1074&bih=499 Do you happen to have any idea where Google is getting this regionalised data from and in particular the pricing which is incorrect? We have a Google (UK) Product Feed however the prices given are different than those being displayed in this localised search. Additionally the product feed that we supply relates to our main website and not a specific store. If you click through to compare prices from multiple merchants you'll see our prices being listed correctly under our company name and website rather than the incorrect pricing attributed to a specific store. I have checked our Google Places Account and our Google Product Feed Account but I just can't figure out where this data and incorrect pricing is coming from and indeed why it only affects our physical stores and not the more generalised website pricing. If someone could point me in the right direction so I can get this corrected I’d appreciate it! Many thanks Chris
Intermediate & Advanced SEO | | ChrisHolgate0 -
Multiple Google+ Local (Google Place) under one email address
As a automotive dealership group, we have 15+ business listings set up under one Google+ local account. Google+ Local (Google Places) offers the ability to upload a data file for 10+ listings, so we've kept all listings under one login for efficiency. Is there any specific local SEO benefit or any general benefit at all to having each business listing set up under their own separate email address?
Intermediate & Advanced SEO | | autoczar0