Should I let Google crawl my production server if the site is still under development?
-
I am building out a brand new site. It's built on Wordpress so I've been tinkering with the themes and plug-ins on the production server. To my surprise, less than a week after installing Wordpress, I have pages in the index.
I've seen advice in this forum about blocking search bots from dev servers to prevent duplicate content, but this is my production server so it seems like a bad idea.
Any advice on the best way to proceed? Block or no block? Or something else? (I know how to block, so I'm not looking for instructions).
- We're around 3 months from officially launching (possibly less).
- We'll start to have real content on the site some time in June, even though we aren't planning to launch.
- We should have a development environment ready in the next couple of weeks.
Thanks!
-
Thank you for the detailed response, Paul. I'll get cracking on your suggestions.
I was mostly worried that if I blocked it now, it would be mad at me later. You've given me a way to deal with the bot concerns.
I am less concerned that anyone will find these pages. I only knew about their index status because of one of my monitoring services which alerted me that google was crawling.
-
Thanks for the confirmation, Dan! Looks like you're up & working early on a Sunday morning
-
In my opinion, no, you definitely should NOT allow the production server to be indexed while it's in this state. For all intents and purposes it IS your dev server at the moment, and the last thing you want is for the search crawlers to think that what's there will be representative of the quality of your site when it's finished.
My recommendation:
- get the current site out of the SERPs. (Use WordPress setting in Settings -> Read to check the "Discourage from indexing" box. DON'T add a no-index in robots.txt until the pages have all dropped out of the SERPs)
- when the dev site goes into operation, make _certain_right from the start it cannot be crawled (vastly better than trying to fix the problem after it get's accidentally indexed).
- as soon as you have time, build a proper front page and a few content pages on the production site that indicate what the full site will be about, and get some strong basic, well-written content on there that will also remain after the go-live. (keep ALL the rest of the pages of the prod site out of the SERPs with meta no-index tags)
- once you have a the new, stable, basic content up on prod, allow the SEs to start indexing it.
This gets the messy stuff out of the SERPs before it can pollute the index (and gives you a bad reputation with any actual visitors to the site who shouldn't be seeing your tinkering). By getting some real content as soon as possible, even on a very basic template, you'll start giving the SEs a quality idea of what is to come. Wouldn't hurt to start building a few backlinks once the basic content is up on prod - e.g. links from its new social profiles etc.
This way, when the full site goes live, you'll already have some quality visibility in the engines, so it will be quicker to get the rest of the new site crawled and indexed.
Does that make sense?
Paul
P.S. If at all appropriate, use the basic prod content to show why/how they should connect with you on social media, and offer them a chance to sign up for your newsletter notification of when the site goes live. (It's never too early to start trying to get those subscribers!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Competitor on same server (so 2 domains in same branche on same server, also same technique)
Hi, To holding of a client of ours, has bought the webshop from a competitor. They have moved the domain of the competitor to their own server, and also changed the technique so both sites have the same CMS and also same technique on the front-end.
Intermediate & Advanced SEO | | Dennis1992038
How bad is this for SEO? Should they change from server ASAP and are there solutions to do stay on the same server but use something like an CDN? Looking forward to your thoughts. Thanks!0 -
Is possible to submit a XML sitemap to Google without using Google Search Console?
We have a client that will not grant us access to their Google Search Console (don't ask us why). Is there anyway possible to submit a XML sitemap to Google without using GSC? Thanks
Intermediate & Advanced SEO | | RosemaryB0 -
Lately I have noticed Google indexing many files on the site without the .html extension
Hello, Our site, while we convert, remains in HTML 4.0. Fle names such as http://www.sample.com/samples/index.shtml are being picked up in the SERPS as http://www.sample.com/samples/ even when I use the "rel="canonical" tag and specify the full file name therein as recommended. The link to the truncated URL (http://www.sample.com/samples/) results in what MOZ shows as fewer incoming links than the full file name is shown as having incoming. I am not sure if this is causing a loss in placement (the MOZ stats are showing a decline of late), which I have seen recently (of course, I am aware of other possible reasons, such as not being in HTML5 yet). Any help with this would be great. Thank you in advance
Intermediate & Advanced SEO | | gheh20130 -
Does Domain Authority remian if site owner and server changes
Hi i am looking at buying an existing domain. It is currently vacant and the site is just on a godaddy holdpage. if i buy this domain, will its DA and PA TF CF still remain if we don't lose the links? thanks
Intermediate & Advanced SEO | | Direct_Ram0 -
Homepage not ranking in Google AU, but ranking in Google UK?
Hey everyone, My homepage has not been ranking for it's primary keyword in Google Australia for many months now. Yesterday when I was using a UK Proxy and searching via Google UK I found my homepage/primary keyword ranked on page 8 in the UK. Now in Australia my website ranks on page 6 but it's for other pages on my website (and it always changes from different page to page). Previously my page was popping up at the bottom of page 1 and page 2. I've been trying many things and waiting weeks to see if it had any impact for over 4 months but I'm pretty lost for ideas now. Especially after what I saw yesterday in Google UK. I'd be very grateful if someone has had the same experience of suggestions and what I should try doing. I did a small audit on my page and because the site is focused on one product and features the primary keyword I took steps to try and fix the issue. I did the following: I noticed the developer had added H1 tags to many places on the homepage so I removed them all to make sure I wasn't getting an over optimization penalty. Cleaned up some of my links because I was not sure if this was the issue (I've never had a warning within Google webmaster tools) Changed the title tags/h tags on secondary pages not to feature the primary keyword as much Made some pages 'noindex' to try and see if this would take away the emphases on the secondary pages Resubmitted by XML sitemaps to Google Just recently claimed a local listings place in Google (still need to verify) and fixed up citations of my address/phone numbers etc (However it's not a local business - sells Australia wide) Added some new backlinks from AU sites (only a handful though) The only other option I can think of is to replace the name of the product on secondary pages to a different appreciation to make sure that the keyword isn't featured there. Some other notes on the site: When site do a 'site:url' search my homepage comes up at the top The site sometimes ranked for a secondary keyword on the front page in specific locations in Australia (but goes to a localised City page). I've noindexed these as a test to see if something with localisation is messing it around. I do have links from AU but I do have links from .com and wherever else. Any tips, advice, would be fantastic. Thanks
Intermediate & Advanced SEO | | AdaptDigital0 -
Unsure where Google has sourced this inaccurate Product Data
Hi, This is a slightly odd one I was hoping someone could shed some light on. One of our staff just did a Google search and located these listings on Google UK Product Search: http://www.google.co.uk/search?q=ink+cartridges&hl=en&sa=X&biw=1074&bih=499&tbm=shop&prmd=imvns#q=ink+cartridges&hl=en&sa=X&tbs=store:3287803270081455254&tbm=shop&prmd=imvns&ei=xp5pUP6uN8i_0QXUuoHADQ&ved=0CI0BEMcMMAE&bav=on.2,or.r_gc.r_pw.r_qf.&fp=333b49ec245f6031&biw=1074&bih=499 Do you happen to have any idea where Google is getting this regionalised data from and in particular the pricing which is incorrect? We have a Google (UK) Product Feed however the prices given are different than those being displayed in this localised search. Additionally the product feed that we supply relates to our main website and not a specific store. If you click through to compare prices from multiple merchants you'll see our prices being listed correctly under our company name and website rather than the incorrect pricing attributed to a specific store. I have checked our Google Places Account and our Google Product Feed Account but I just can't figure out where this data and incorrect pricing is coming from and indeed why it only affects our physical stores and not the more generalised website pricing. If someone could point me in the right direction so I can get this corrected I’d appreciate it! Many thanks Chris
Intermediate & Advanced SEO | | ChrisHolgate0 -
2 sites or one sites: 2 locations
Hello, I have a dog training client who is offering services in 2 separate locations. We're looking to be first in the non-local search results and also rank well in google places. Would it be better to go for 2 separate sites or one site and try to rank for 2 different locations with one site? There's both local and standard search results when we type in our keywords. Thanks!
Intermediate & Advanced SEO | | BobGW0 -
How to handle a server outage if I have two sites
I operate a web application. It consists of two sites, www.mysite.com and app.mysite.com. As you might imagine, www is used for marketing purposes, and it's our main organic search entry point. The app.mysite.com domain is where our application portal is for customers, and it is also where our login and registration pages are located. Currently, www.mysite.com is experiencing a catastrophic outage and is returning 504 errors, but app.mysite.com is on a totally separate system with a lot redundancy, and is doing just fine. If we get traffic from referrals or search, we want that traffic to be able to login and register, so we've replaced the 504 error with a 302 redirect to app.mysite.com until the situation is resolved. This provides the best possible experience for users (nothing's worse than a 504). How will this affect SEO? Is there something other than a 302 that I should be doing with the broken www.mysite.com domain?
Intermediate & Advanced SEO | | Ehren0