New server update + wrong robots.txt = lost SERP rankings
-
Over the weekend, we updated our store to a new server. Before the switch, we had a robots.txt file on the new server that disallowed its contents from being indexed (we didn't want duplicate pages from both old and new servers).
When we finally made the switch, we somehow forgot to remove that robots.txt file, so the new pages weren't indexed. We quickly put our good robots.txt in place, and we submitted a request for a re-crawl of the site.
The problem is that many of our search rankings have changed. We were ranking #2 for some keywords, and now we're not showing up at all. Is there anything we can do? Google Webmaster Tools says that the next crawl could take up to weeks! Any suggestions will be much appreciated.
-
Dr. Pete,
I just ran across one of your webinars yesterday and you brought up some great ideas. Earned a few points in my book
Too often SEOs see changes in the rankings and react to counter-act the change. Most of the time these bounces are actually a GOOD sign. It means Google saw your changes and is adjusting to them. If your changes were positive you should see positive results. I have rarely found an issue where a user made a positive change and got a negative result from Google. Patience is a virtue.
-
Thanks everyone for the help! Fortunately we remedied the problem almost immediately, so it only took about a day to get our rankings back. I think the sitemap and fixed robots.txt were the most important factors.
-
I agree, let Google re-index first and then re evaluate the situation.
-
I hate to say it, but @inhouseninja is right - there's not a lot you can do, and over-reacting could be very dangerous. In other words - don't make a ton of changes just to offset this - Google will re-index.
A few minor cues that are safe:
(1) Re-submit your XML sitemap
(2) Build a few new links (authoritative ones, especially)
(3) Hit social media with your new URLs
All 3 are at least nudges to re-index. They aren't magic bullets, but you need to get Google's attention.
-
Remain calm. You should be just fine. It just takes time for Google to digest the new robots.txt. I would be concerned if things didn't change in 3-4 weeks. Adopt a rule to not freak out on Google until you've given the problem 14 days to resolve. Sometimes Google moves things around and this is natural.
If you want Google to crawl your site faster, build some links and do some social media. That will encourage Google to speed it up.
-
If this is all that happened the next crawl should fix it. Just sit tight and they should bounce up again in a week or so.
-
That does not sound fun at all.... So you just changed the server, complete copy?
My first question would be other than the server did anything else change? Copy or URL's
My second question would be is the other server still up and live to the internet?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New blog on a separate server to the main website?
We have a potential client who operates a jobs board in a niche sector in the UK. They want to start a blog but don't want to set it up on the same server as the main jobs site. Discussion started around Wordpress, and their preference is for the WP.com hosted version in a directory or subdomain of the TLD. Our concerns are around the different locations of the two sites (impact of two diff server locations and IP addresses?) but also the limitation of WP.com to interlink the two sites enough that they provide a decent customer experience. Thoughts, musings, advice - all welcome! Tks
Intermediate & Advanced SEO | | AB-Marketing0 -
User intent and ranking
Hello, I was doing search to see who ranks on the keyword "bike tours" i noticed a lot of website rank without content "text". Is it because they fulfil the user intent with a search box where you can search by date and destination, trip type and price that google ranks those webpages ? It is the same for the keyword Paris bike tours https://www.fattiretours.com/paris How do they rank with so little content (once again is it because the fulfil the user intent ) Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
Robots.txt Help
I need help to create robots.txt file. Please let me know what to add in the file. any real example or working example.?
Intermediate & Advanced SEO | | Michael.Leonard0 -
My SERP Rankings are unbelievably inconsistent, how is it possible
Almost all of the keywords i track in moz's keywords ranking report show insane inconsistency. nearly all my keywords (according to moz's rank tracker history) were in the first or second rank for a week or 2 during the last 6 month.
Intermediate & Advanced SEO | | datawork
nearly all my keywords were nowhere in the top 5 pages in the weeks prior or after the week that the keyword was ranked at the top of the first page. I'm adding a image of an excel chart showing the ranking - each row is a different keyword, each column is a different date. 51 means 51+ (or - nowhere to be found in the first top pages) Could you please help me make sense out of this? How can it change so frantically? Thanks, eZDVEEW.png0 -
Updated My Business Profiles & Still Not Ranking in Local SEO... What Next?
I have used Get Listed and a few other services to update my profiles for my company Health Care Associates (its a home health care agency). http:healthcareassociates.net. I have added pictures, categories, descriptions, key words, etc. and it doesn't seem to help. We are still not ranking in the search engines for "home care grand rapids", "home health care" etc. What else can I do to optimize the local search? Todd
Intermediate & Advanced SEO | | t1kuslik0 -
Very Puzzled --- 301 ReDirects Did Not Work - Lost Rankings - Any Thoughts?
This one has us stumped and frustrated, hopefully someone out there in SEOMoz land can give us some thoughts and/or suggestions on what's going on and how to remedy. This is a follow-up to a post I made awhile back. Here is an excerpt from the original post -- We currently have 3 different versions of our State Business-for-Sale listings pages - the versions are: Version 1 -- Preferred Version (Links on Homepage www.businessbroker.net) http://www.businessbroker.net/State/Vermont-Businesses_For_Sale.aspx Title = Vermont Business for Sale Ads - Vermont Businesses for Sale & Business Brokers - Sell a Business on Business Broker (I realize the title needs work) Version 2: (Links on this page: http://www.businessbroker.net/listings/blistings.ihtml) URL Prior to 301 change --- http://www.businessbroker.net/Businesses_For_Sale-State-Vermont.aspx Title = Vermont Business for Sale | 120 Vermont Businesses for Sale | BusinessBroker.net Version 3: (Links on this page: http://www.businessbroker.net/businessesforsale.ihtml) URL Prior to 301 change --- http://www.businessbroker.net/listings/business_for_sale_vermont.ihtml Title = Vermont Businesses for Sale at BusinessBroker.net - Vermont Business for Sale While the page titles and meta data are a bit different, the bulk of the page content (which is the listings rendered) are identical. OK, so we decided to test this on 5 of our State pages - I will use VERMONT in this discussion. We did 301 ReDirects on Version 2 and Version 3 -- they now redirect to Version 1 - we did the redirects and also changed the URL's on the pages. Prior to the change, we were ranking for keywords like "Vermont Business for Sale" and some other similar keywords -- on 1st page of Google --- now, we have lost our rankings big time. Did we do something wrong? I thought when you did 301's the majority of link juice was supposed to be preserved (losing 10% or so) -- this didn't happen in our case. Any help on what we can do would be appreciated. We only did 5 States as a test and also noticed big drops for Maine as well. These were both states where VERSION 2 was the page that was showing up in SERPs. Thanks in advance for wading through this long post and any help you can provide!! Matt
Intermediate & Advanced SEO | | MWM37720 -
301 redirect or Robots.txt on an interstatial page
Hey guys, I have an affiliate tracking system that works like this : an affiliate puts up a certain code on his site, for example : www.domain.com/track/aff_id This url leads to a page where the hit is counted, analysed and then 302 redirects to my sales page with the affiliates ID in the url : www.mysalespage.com/?=aff_id. However, we've noticed recently that one affiliate seems to be ranking for our own name and the url google indexed was his tracking url (domain.com/track/aff_id). Which is strange because there is absolutely nothing on that page, its just an interstatial page so that our stats tracking software can properly filter hits. To remove the affiliate's url from showing up in the serps, I've come up with 2 solutions : 1 - Change the redirect to a 301 redirect on his track page. 2 - Change our robots.txt page to block all domain.com/track/ pages from being indexed. My question is : if I 301 redirect instead of 302, will I keep the affiliates from outranking me for my own name AND pass on link juice or should I simply block google from crawling the interstatial tracking pages?
Intermediate & Advanced SEO | | CrakJason0 -
Quick ranking boost
Hi, What is hte quickest way to get up the rankings for a particular not hugley competitive keyword ? Thanks,
Intermediate & Advanced SEO | | Prongo0