New server update + wrong robots.txt = lost SERP rankings
-
Over the weekend, we updated our store to a new server. Before the switch, we had a robots.txt file on the new server that disallowed its contents from being indexed (we didn't want duplicate pages from both old and new servers).
When we finally made the switch, we somehow forgot to remove that robots.txt file, so the new pages weren't indexed. We quickly put our good robots.txt in place, and we submitted a request for a re-crawl of the site.
The problem is that many of our search rankings have changed. We were ranking #2 for some keywords, and now we're not showing up at all. Is there anything we can do? Google Webmaster Tools says that the next crawl could take up to weeks! Any suggestions will be much appreciated.
-
Dr. Pete,
I just ran across one of your webinars yesterday and you brought up some great ideas. Earned a few points in my book
Too often SEOs see changes in the rankings and react to counter-act the change. Most of the time these bounces are actually a GOOD sign. It means Google saw your changes and is adjusting to them. If your changes were positive you should see positive results. I have rarely found an issue where a user made a positive change and got a negative result from Google. Patience is a virtue.
-
Thanks everyone for the help! Fortunately we remedied the problem almost immediately, so it only took about a day to get our rankings back. I think the sitemap and fixed robots.txt were the most important factors.
-
I agree, let Google re-index first and then re evaluate the situation.
-
I hate to say it, but @inhouseninja is right - there's not a lot you can do, and over-reacting could be very dangerous. In other words - don't make a ton of changes just to offset this - Google will re-index.
A few minor cues that are safe:
(1) Re-submit your XML sitemap
(2) Build a few new links (authoritative ones, especially)
(3) Hit social media with your new URLs
All 3 are at least nudges to re-index. They aren't magic bullets, but you need to get Google's attention.
-
Remain calm. You should be just fine. It just takes time for Google to digest the new robots.txt. I would be concerned if things didn't change in 3-4 weeks. Adopt a rule to not freak out on Google until you've given the problem 14 days to resolve. Sometimes Google moves things around and this is natural.
If you want Google to crawl your site faster, build some links and do some social media. That will encourage Google to speed it up.
-
If this is all that happened the next crawl should fix it. Just sit tight and they should bounce up again in a week or so.
-
That does not sound fun at all.... So you just changed the server, complete copy?
My first question would be other than the server did anything else change? Copy or URL's
My second question would be is the other server still up and live to the internet?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ranking Issue for New Site
Hi all, I have got a specific SEO challenge. 6 months ago, we started to build an eCommerce site (located in the UK). In order to speed up the site launch, we copied the entire site over from an existing site based in Ireland. Now, the new UK site has been running for 5 months. Google has indexed many pages, which is good, but we can't rank high (position: between 20-30 for most pages). We thought it was because of content duplication in spite of different regions. So we tried to optimize the pages for the UK site to make them more UK-related and avoid content duplication. I've also used schema to tell google it's a UK-based site and set up Google my business and got more local citations. Besides, If you could give me any suggestions, it'd be perfect.
Intermediate & Advanced SEO | | Insightful_Media
Thank you so much for your time and advice.1 -
Ranking all but lost after new content pushed
Okay, I thought I was following best practices. In our industry, electronic hardware, we were ranking well for a particular product line (/spacers) but we wanted to do better. We addressed several concerns that Moz found first; duplicate page titles, lack of meta-descriptions and overall lack of targeted keywords. We also took a new approach to add a better structure to our site. Instead of being presented with a list of part numbers we wanted the user to learn more about our products with content. So we added a /products page with content and a product specific page (/spacers) that is almost a definitive buyers guide. We are attempting to answer the questions that we think our customers find most relevant. Well our customers might find it relevant but Google sure didn't. After our deployment of new content our rankings for targeted keywords in Google fell from 10-15 to 80-95 As an open ended question, could somebody explain to me why our ranks fell off a cliff? Homepage: https://www.lyntron.com
Intermediate & Advanced SEO | | jandk4014
New catalog summary page: https://www.lyntron.com/products
New content with focus to rank high: https://www.lyntron.com/spacers TPdn6ym1 -
New blog on a separate server to the main website?
We have a potential client who operates a jobs board in a niche sector in the UK. They want to start a blog but don't want to set it up on the same server as the main jobs site. Discussion started around Wordpress, and their preference is for the WP.com hosted version in a directory or subdomain of the TLD. Our concerns are around the different locations of the two sites (impact of two diff server locations and IP addresses?) but also the limitation of WP.com to interlink the two sites enough that they provide a decent customer experience. Thoughts, musings, advice - all welcome! Tks
Intermediate & Advanced SEO | | AB-Marketing0 -
How will canonicalizing an https page affect the SERP-ranked http version of that page?
Hey guys, Until recently, my site has been serving traffic over both http and https depending on the user request. Because I only want to serve traffic over https, I've begun redirecting http traffic to https. Reviewing my SEO performance in Moz, I see that for some search terms, an http page shows up on the SERP, and for other search terms, an https page shows. (There aren't really any duplicate pages, just the same pages being served on either http or https.) My question is about canonical tags in this context. Suppose I canonicalize the https version of a page which is already ranked on the SERP as http. Will the link juice from the SERP-ranked http version of that page immediately flow to the now-canonical https version? Will the https version of the page immediately replace the http version on the SERP, with the same ranking? Thank you for your time!
Intermediate & Advanced SEO | | JGRLLC0 -
Robots.txt - blocking JavaScript and CSS, best practice for Magento
Hi Mozzers, I'm looking for some feedback regarding best practices for setting up Robots.txt file in Magento. I'm concerned we are blocking bots from crawling essential information for page rank. My main concern comes with blocking JavaScript and CSS, are you supposed to block JavaScript and CSS or not? You can view our robots.txt file here Thanks, Blake
Intermediate & Advanced SEO | | LeapOfBelief0 -
How to get head office address appear on SERPS under brand serps?
Hi all, We have different offices in UK in different cities. Until last month our main office was appearing on Google results but lately i noticed that our other office is appearing on serps when i search with brand name. We have 3 addresses registered with Google Places. Any ideas how we can get head office to appear on serps under our brand name? Thanks
Intermediate & Advanced SEO | | Rubix0 -
Launching 20 new URLs to branch out from a multi brand website. How do I make sure the rankings stay?
I have a client with a multiple brand website currently. The site has landing pages and directories for each of its 20 brands under one URL. The plan is to build 20 separate brand websites with new domains and have the old content redirect to its corresponding new URL. The problem that I envision is being able to successfully carry all of the existing indexed content over while only using 301 redirects and submitting the new domains to Webmaster tools. Has anyone done this in the past successfully and do you have any recommendations?
Intermediate & Advanced SEO | | Empower_MediaMarketing0 -
Why is noindex more effective than robots.txt?
In this post, http://www.seomoz.org/blog/restricting-robot-access-for-improved-seo, it mentions that the noindex tag is more effective than using robots.txt for keeping URLs out of the index. Why is this?
Intermediate & Advanced SEO | | nicole.healthline0