New server update + wrong robots.txt = lost SERP rankings
-
Over the weekend, we updated our store to a new server. Before the switch, we had a robots.txt file on the new server that disallowed its contents from being indexed (we didn't want duplicate pages from both old and new servers).
When we finally made the switch, we somehow forgot to remove that robots.txt file, so the new pages weren't indexed. We quickly put our good robots.txt in place, and we submitted a request for a re-crawl of the site.
The problem is that many of our search rankings have changed. We were ranking #2 for some keywords, and now we're not showing up at all. Is there anything we can do? Google Webmaster Tools says that the next crawl could take up to weeks! Any suggestions will be much appreciated.
-
Dr. Pete,
I just ran across one of your webinars yesterday and you brought up some great ideas. Earned a few points in my book
Too often SEOs see changes in the rankings and react to counter-act the change. Most of the time these bounces are actually a GOOD sign. It means Google saw your changes and is adjusting to them. If your changes were positive you should see positive results. I have rarely found an issue where a user made a positive change and got a negative result from Google. Patience is a virtue.
-
Thanks everyone for the help! Fortunately we remedied the problem almost immediately, so it only took about a day to get our rankings back. I think the sitemap and fixed robots.txt were the most important factors.
-
I agree, let Google re-index first and then re evaluate the situation.
-
I hate to say it, but @inhouseninja is right - there's not a lot you can do, and over-reacting could be very dangerous. In other words - don't make a ton of changes just to offset this - Google will re-index.
A few minor cues that are safe:
(1) Re-submit your XML sitemap
(2) Build a few new links (authoritative ones, especially)
(3) Hit social media with your new URLs
All 3 are at least nudges to re-index. They aren't magic bullets, but you need to get Google's attention.
-
Remain calm. You should be just fine. It just takes time for Google to digest the new robots.txt. I would be concerned if things didn't change in 3-4 weeks. Adopt a rule to not freak out on Google until you've given the problem 14 days to resolve. Sometimes Google moves things around and this is natural.
If you want Google to crawl your site faster, build some links and do some social media. That will encourage Google to speed it up.
-
If this is all that happened the next crawl should fix it. Just sit tight and they should bounce up again in a week or so.
-
That does not sound fun at all.... So you just changed the server, complete copy?
My first question would be other than the server did anything else change? Copy or URL's
My second question would be is the other server still up and live to the internet?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Lost ALL my rankings
Dear experts, We have suddenly lost all our positions on our keywords, I’m talking for example from position 2 to 35 in one huge drop ☹ . Even when you search for our name and domain name namely “KV16”, the main page does NOT come up. (We do however still get the local business / map result on the right side!) I have looked in google search console, but nothing to see here – no penalty. However, I need to mention, that we just now, after the issue, setup the google search console – so maybe that is why there is no information about a penalty!? (If the problem is a penalty, btw we have not been doing any link building!) One year ago, we made a permanent redirect from our old site, www.kontoret-gammeltorv.dk, because we moved our coworking office, to a new location, and needed a new name – (because the old name is referring to a location, hence no longer made sense.) This worked flawlessly, and we got all the link juice, and the new site ranked just as good as the old one. Aprox four months ago, we made some changes to the site, where we amongst other things, changed the title description on the frontpage, focusing more on another keyword – and doing this we unknowingly change the permalink to the frontpage (in WordPress). I have read that this can do some real damage, however its so long ago, that it seems unlikely to be the cause!? Please help!? Thanks a million, in advance. Cheers, Christian pJp4Uyx
Intermediate & Advanced SEO | | Christian_T0 -
Robots.txt blocked internal resources Wordpress
Hi all, We've recently migrated a Wordpress website from staging to live, but the robots.txt was deleted. I've created the following new one: User-agent: *
Intermediate & Advanced SEO | | Mat_C
Allow: /
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Allow: /wp-admin/admin-ajax.php However, in the site audit on SemRush, I now get the mention that a lot of pages have issues with blocked internal resources in robots.txt file. These blocked internal resources are all cached and minified css elements: links, images and scripts. Does this mean that Google won't crawl some parts of these pages with blocked resources correctly and thus won't be able to follow these links and index the images? In other words, is this any cause for concern regarding SEO? Of course I can change the robots.txt again, but will urls like https://example.com/wp-content/cache/minify/df983.js end up in the index? Thanks for your thoughts!2 -
How to rank a product page
Hello, I have a product page and just did a little introduction ( 5 lines of text) but I have the feeling this is not enough for google. What is the solution ? Is it to increase the amount of content even though it is not going to be user friendly ? Or will google look at the product page I link too and take into account the content on those subpages to boost my "pillar / product page". Thank you,
Intermediate & Advanced SEO | | seoanalytics1 -
Robots txt is case senstive? Pls suggest
Hi i have seen few urls in the html improvements duplicate titles Can i disable one of the below url in the robots.txt? /store/Solar-Home-UPS-1KV-System/75652
Intermediate & Advanced SEO | | Rahim119
/store/solar-home-ups-1kv-system/75652 if i disable this Disallow: /store/Solar-Home-UPS-1KV-System/75652 will the Search engines scan this /store/solar-home-ups-1kv-system/75652 im little confused with case senstive.. Pls suggest go ahead or not in the robots.txt0 -
Robots.txt question
I notice something weird in Google robots. txt tester I have this line Disallow: display= in my robots.text but whatever URL I give to test it says blocked and shows this line in robots.text for example this line is to block pages like http://www.abc.com/lamps/floorlamps?display=table but if I test http://www.abc.com/lamps/floorlamps or any page it shows as blocked due to Disallow: display= am I doing something wrong or Google is just acting strange? I don't think pages with no display= are blocked in real.
Intermediate & Advanced SEO | | rbai0 -
Impact of May 2015 quality update and July Panda update on specialty brands or niche retail
We are seeing the following trend in our rankings and traffic after the recent Google algorithm updates (May 2015 quality/phantom, and July 2015 Panda), and I am curious if anyone here has encountered similar and/or has any good ideas on how to react. Background - we operate in a niche segment, but compete for keywords with large home improvement stores and mass retailers. In the past, prior to May 2015, we generally ranked higher than the large home improvement stores and mass retailers for our key specific terms in our niche. We believed that it was because we have a very specialized focus and so our store was highly relevant for someone searching in that niche (for example for the name of the product category as a keyword). In general, we ranked #1-3. Along with a few of our competitors in our niche. And then would be the big box home improvement stores in spots 5-10. The change we saw starting in May is that now all the home improvement stores and also a few large multi-category retailers took over those top 5 spots and bumped all the specialty retailers and the specialty brand manufacturers (like us) down. Our direct competitors in our niche all seem to have been impacted pretty much the same as us. So, in summary it seems like these latest updates may have favored the more general retailers but with stronger domain authority than the more specific but smaller retailers. Hard to know for sure, but this is the trend we believe we see. So, that said, what are some good strategies to respond to this situation? We can't really compete on overall domain authority with these huge retailers. And our previously successful strategy of having a very focused niche, with lots of helpful content, videos, instructional guides, etc. no longer seems to be enough. Has anyone else seen similar results since this past May? Where specialty retail or brand sites lost ground to larger general retailers? And if so, has anyone found any good strategies to gain back their previous rankings, or at least partially?
Intermediate & Advanced SEO | | drewk1 -
Files blocked in robot.txt and seo
I use joomla and I have blocked the following in my robots.txt is there anything that is bad for seo ? User-agent: * Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /images/ Disallow: /includes/ Disallow: /installation/ Disallow: /language/ Disallow: /libraries/ Disallow: /media/ Disallow: /modules/ Disallow: /plugins/ Disallow: /templates/ Disallow: /tmp/ Disallow: /xmlrpc/ Disallow: /mailto:myemail@myemail.com/ Disallow: /javascript:void(0) Disallow: /.pdf
Intermediate & Advanced SEO | | seoanalytics0 -
Robots.txt & url removal vs. noindex, follow?
When de-indexing pages from google, what are the pros & cons of each of the below two options: robots.txt & requesting url removal from google webmasters Use the noindex, follow meta tag on all doctor profile pages Keep the URLs in the Sitemap file so that Google will recrawl them and find the noindex meta tag make sure that they're not disallowed by the robots.txt file
Intermediate & Advanced SEO | | nicole.healthline0