Site being indexed by Google before it has launched
-
We are currently coming towards the end of migrating one of our retail sites over to magento. To our horror, we find out today that some pages are already being indexed by Google, and we have started receiving orders through new site. Do you have any suggestions for what may have caused this? Or similarly, what the best solution would be to de-index ourselves? We most recently excluded anything with a certain parameter from robots.txt - could this being implemented incorrectly have caused this issue?
Thanks
-
Google has ignored a robots.txt for my site because it thought we were "hiding" important content.
Depending on your site structure, putting a "no index" tag on your page is also a good idea. Doing that as pages are built can prevent it (better) from being indexed in the first place. Then you remove the tag when you are ready to go live. And, submit your updated sitemap via GWT when you finish your move so Google will know what pages to index.
-
I would be very careful on how you manage this right now. It all depends on the old and new urls. If this is something in Dev staging, like http://dev.examplesite.com, then it is alright to do everything possible to clean them up. But if the new urls are going to be the new final urls, then you have to be careful.
The best thing you could do right now would be to password protect the new site. That way there isn't a chance of anything getting to it.
All it takes is Google finding one link to the site then it will crawl from there.
-
Exclude the whole new site using robots.txt, and remove the current new site-indexed URL's within Webmaster Tools: Optimization > Remove URL's.
When the new site is ready to launch, undo these removed URL's using the "Reinclude" button, also found within the Remove URL's section of Webmaster Tools. Do this prior to 301'ing the old site pages to the new, and you will probably wish to announce the new site launch via social media as well to encourage indexation. +1'ing a page is considered the fastest way to get it indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website homepage temporarily getting removed from google index
hi, website: www.snackmagic.com The home page goes out of google index for some hours and then comes back. We are not sure why our home page is getting de-indexed temporarily. This doesn't happen with other pages on our website. This has been happening intermittently in the gap of 2-3 days. Any inputs will be very useful for us to debug this issue Thanks
Technical SEO | | manikbystadium0 -
Google indexing .com and .co.uk site
Hi, I am working on a site that is experiencing indexation problems: To give you an idea, the website should be www.example.com however, Google seems to index www.example.co.uk as well. It doesn’t seem to honour the 301 redirect that is on the co.uk site. This is causing quite a few reporting and tracking issues. This happened the first time in November 2016 and there was an issue identified in the DDOS protection which meant we would have to point www.example.co.uk to the same DNS as www.example.com. This was implemented and made no difference. I cleaned up the htaccess file and this made no difference either. In June 2017, Google finally indexed the correct URL, but I can’t be sure what changed it. I have now migrated the site onto https and www.example.co.uk has been reindexed in Google alongside www.example.com I have been advised that the http needs to be removed from DDOS which is in motion I have also redirected http://www.example.co.uk straight to https://www.example.com to prevent chain redirects I can’t block the site via robot.txt unless I take the redirects off which could mean that I lose my rankings. I should also mention that I haven't actually lost any rankings, it's just replaced some URLs with co.uk and others have remained the same. Could you please advise what further steps I should take to ensure the correct URL’s are indexed in Google?
Technical SEO | | Niki_10 -
My video sitemap is not being index by Google
Dear friends, I have a videos portal. I created a video sitemap.xml and submit in to GWT but after 20 days it has not been indexed. I have verified in bing webmaster as well. All videos are dynamically being fetched from server. My all static pages have been indexed but not videos. Please help me where am I doing the mistake. There are no separate pages for single videos. All the content is dynamically coming from server. Please help me. your answers will be more appreciated................. Thanks
Technical SEO | | docbeans0 -
How to Let Google Know I am a new Site Owner and to Remove or De-value all backlinks?
I am looking to buy a new domain for a brand. Problem is the domain has was registered in 1996 and has around 6k backlinks (according to ahrefs) that I need removed as the old content will have no relevance to my new site. Should I just disavow all of them? Is there anything "special" I can do to let Google know that it will be new site owner/content and to remove/discount the current links?
Technical SEO | | RichardSEO0 -
Searching in Google using the Site:www.example.com specification - is it in an order?
Hi Gurus, Just a quick searching question. If you do a Google search using the site: specification eg. site:www.example.com Is the list returned by Google in an order of something similar to 'Page Authority' or some other order eg. page first seen date etc. Because you are looking at your single site, is Google listing your pages back to you in it's perceived order of current 'popularity'? Thanks, Brad
Technical SEO | | BM70 -
Odd Google Indexing Issue
I have encountered something odd with Google indexing. According to the Google cache my site was last updated on April 6. I had been making a series of changes on April 7th and none of them show up in the cached version of the site (naturally). Then, on the 8th, my rankings seem to have dropped about 6 places and the main SERP is showing a text that isn't even on the Web site. The cached version has the correct page title from the page that was indexed on the 6th. How do I learn where Google is picking this up from? There is a clean page title tag on my Web site. I've checked the server, etc to see what's going on. The text isn't completely unrelated, but it definitely impacted my ranking. Does Google ever have these hiccups when indexing?
Technical SEO | | VERBInteractive0 -
Partial Site Move -- Tell Google Entire Site Moved?
OK this one's a little confusing, please try to follow along. We recently went through a rebranding where we brought a new domain online for one of our brands (we'll call this domain 'B' -- it's also not the site linked to in my profile, not to confuse things). This brand accounted for 90% of the pages and 90% of the e-comm on the existing domain (we'll call the existing domain 'A') . 'A' was also redesigned and it's URL structure has changed. We have 301s in place on A that redirect to B for those 90% of pages and we also have internal 301s on A for the remaining 10% of pages whose URL has changed as a result of the A redesign What I'm wondering is if I should tell Google through webmaster tools that 'A' is now 'B' through the 'Change of Address' form. If I do this, will the existing products that remain on A suffer? I suppose I could just 301 the 10% of URLs on B back to A but I'm wondering if Google would see that as a loop since I just got done telling it that A is now B. I realize there probably isn't a perfect answer here but I'm looking for the "least worst" solution. I also realize that it's not optimal that we moved 90% of the pages from A to B, but it's the situation we're in.
Technical SEO | | badgerdigital0 -
My site went from #3 to #2 to #1 to #10 in Google.nl
I have a site which has been #3 in Google.nl for a certain long tail phrase for pretty long. After I optimized my site some more for this specific phrase I went to #2 and eventually some weeks later to #1. But when I finally became #1, the celebration was short, as two weeks thereafter I dropped to #10 where I have been stuck for several weeks now... Can this be some sort of over optimization penalty? I personally don't think so as I hardly created links with the anchor, or a variation of it. On page also seems pretty much in order to me. Also, the site is on a exact match domain and used to rank for it, but doesn't even rank in the top #100 anymore. Any advice? PS: I rather don't publish the URL, but if anyone can help out I have no problem sending it in a private message, including the phrase in question.
Technical SEO | | VisualSense0