Site being indexed by Google before it has launched
-
We are currently coming towards the end of migrating one of our retail sites over to magento. To our horror, we find out today that some pages are already being indexed by Google, and we have started receiving orders through new site. Do you have any suggestions for what may have caused this? Or similarly, what the best solution would be to de-index ourselves? We most recently excluded anything with a certain parameter from robots.txt - could this being implemented incorrectly have caused this issue?
Thanks
-
Google has ignored a robots.txt for my site because it thought we were "hiding" important content.
Depending on your site structure, putting a "no index" tag on your page is also a good idea. Doing that as pages are built can prevent it (better) from being indexed in the first place. Then you remove the tag when you are ready to go live. And, submit your updated sitemap via GWT when you finish your move so Google will know what pages to index.
-
I would be very careful on how you manage this right now. It all depends on the old and new urls. If this is something in Dev staging, like http://dev.examplesite.com, then it is alright to do everything possible to clean them up. But if the new urls are going to be the new final urls, then you have to be careful.
The best thing you could do right now would be to password protect the new site. That way there isn't a chance of anything getting to it.
All it takes is Google finding one link to the site then it will crawl from there.
-
Exclude the whole new site using robots.txt, and remove the current new site-indexed URL's within Webmaster Tools: Optimization > Remove URL's.
When the new site is ready to launch, undo these removed URL's using the "Reinclude" button, also found within the Remove URL's section of Webmaster Tools. Do this prior to 301'ing the old site pages to the new, and you will probably wish to announce the new site launch via social media as well to encourage indexation. +1'ing a page is considered the fastest way to get it indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should 'View All' filters on ecommerce sites be indexable?
Hi, I’m looking at a site at the moment that has a lot of products. For some of their category pages they have a ‘View All’ feature available. The URL uses this structure: domain.com/category/sub-category/product domain.com/category/sub-category/view-all < currently noindex applied Should the view all page be available for indexing? The individual sub-categories and products are indexable My immediate reaction is no, so long as the individual sub-cats are?
Technical SEO | | daniel-brooks0 -
My site Has Penalized By google Search Result Without Any Spam Score.
I Recently Make a Site Gizmocombot.com. tHE aITE has NO spam Record NO lousy BACKLINK.it has all unique article can anyone tell us how we can unpenalized our site from google webmaster and google search Result. i attcead a screenshot as well yoou need. 3nzmALp
Technical SEO | | litoginamaaba3332 -
How to check if an individual page is indexed by Google?
So my understanding is that you can use site: [page url without http] to check if a page is indexed by Google, is this 100% reliable though? Just recently Ive worked on a few pages that have not shown up when Ive checked them using site: but they do show up when using info: and also show their cached versions, also the rest of the site and pages above it (the url I was checking was quite deep) are indexed just fine. What does this mean? thank you p.s I do not have WMT or GA access for these sites
Technical SEO | | linklander0 -
How GOOGLE can re-index my site as possible as?
I have facing the question about re-indexing in the google search engine, the case is: i have changed my site meta description but google indexed display part description why?? my site is http://www.green-lotus-trekking.com/everest-base-camp-trek/ whats the problem in meta tag description? Please let me know about this?
Technical SEO | | agsln0 -
Bing indexing at a tiny fraction of Google
I've read through other posts about this but I can't find a solution that works for us. My site is porch.com, 1M+ pages indexed on Google, ~10k on Bing. I've submitted the same sitemaps, and there's nothing different for each bot in our robots file. It looks like Bing is more concerned with our 500 errors than Google, but not sure if that might be causing the issue. Can anyone point me to the right things to be researching/investigating? Fixing errors, sitemap crawling issues, etc. I'm not sure what to spend my time looking into...
Technical SEO | | Porch0 -
Google using descriptions from other websites instead of site's own meta description
In the last month or so, Google has started displaying a description under links to my home page in its search results that doesn't actually come from my site. I have a meta description tag in place and for a very limited set of keywords, that description is displayed, but for the majority of results, it's displaying a description that appears on Alexa.com and a handful of other sites that seem to have copied Alexa's listing, e.g. similarsites.com. The problem is, the description from these other sites isn't particularly descriptive and mentions a service that we no longer provide. So my questions are: Why is Google doing this? Surely that's broken behaviour. How do I fix it?
Technical SEO | | antdesign0 -
A site is not being indexed by Google Yahoo or Bing
This site - http://adoptionconnection.org/ is not being indexed by any of the search engines. I checked the easy stuff - robots text is: <meta name="<a class="attribute-value">robots</a>" content="<a class="attribute-value">all, index, follow</a>" /> <meta name="<a class="attribute-value">robots</a>" content="<a class="attribute-value">noodp</a>" /> <meta name="<a class="attribute-value">robots</a>" content="<a class="attribute-value">noydir</a>" /> I have checked what I can determine would cause the issue but have found nothing to prevent it from being indexed. I'm thinking it may be re-directs etc. Any answer would be great. Thanks in advance,
Technical SEO | | Intergen0 -
Can JavaScrip affect Google's index/ranking?
We have changed our website template about a month ago and since then we experienced a huge drop in rankings, especially with our home page. We kept the same url structure on entire website, pretty much the same content and the same on-page seo. We kind of knew we will have a rank drop but not that huge. We used to rank with the homepage on the top of the second page, and now we lost about 20-25 positions. What we changed is that we made a new homepage structure, more user-friendly and with much more organized information, we also have a slider presenting our main services. 80% of our content on the homepage is included inside the slideshow and 3 tabs, but all these elements are JavaScript. The content is unique and is seo optimized but when I am disabling the JavaScript, it becomes completely unavailable. Could this be the reason for the huge rank drop? I used the Webmaster Tolls' Fetch as Googlebot tool and it looks like Google reads perfectly what's inside the JavaScrip slideshow so I did not worried until now when I found this on SEOMoz: "Try to avoid ... using javascript ... since the search engines will ... not indexed them ... " One more weird thing is that although we have no duplicate content and the entire website has been cached, for a few pages (including the homepage), the picture snipet is from the old website. All main urls are the same, we removed some old ones that we don't need anymore, so we kept all the inbound links. The 301 redirects are properly set. But still, we have a huge rank drop. Also, (not sure if this important or not), the robots.txt file is disallowing some folders like: images, modules, templates... (Joomla components). We still have some html errors and warnings but way less than we had with the old website. Any advice would be much appreciated, thank you!
Technical SEO | | echo10