Longevity of robot.txt files on Google rankings
-
This may be a difficult question to answer without a ton more information, but I'm curious if there's any general thought that could shed some light on the following scenario I've recently heard about and wish to be able to offer some sound advice:
An extremely reputable non-profit site with excellent ranking had gone through a re-design and change-over into WordPress. A robots.txt file was used during development on the dev site on the dev server.
Two months later it was noticed through GA that traffic was way down to the site. It was then discovered that the robot.txt file hadn't been removed and the new site (same content, same nav) went live with it in place. It was removed and a site index forced. How long might it take for the site to re-appear and regain past standing in the SERPs if rankings have been damaged. What would the expected recovery time be?
-
They were paying attention to GA but lapsed and when they checked back in, saw a drop in traffic. Great point about that "critical" message.. The developers did force a crawl and I'm hoping you are correct about the time it might take.
-
Thank you methodicalweb. Great suggestions.
-
Thanks, Travis. You've offered a lot of very interesting points.
I will double-check that they have looked at the server log files, but I'm pretty confident that they have done that.
They did assure me that the proper redirects were done but I'm not sure what they did regarding extensions. There was also a server change.....
-
Thanks for clarifying KeriMorgret. Much appreciated. As are all your thoughts. I will definitely suggest that the monitoring software be used to avoid any future problems. This was such an unnecessary and frustrating experience.
-
If they were paying attention to WMT they would have seen a "critical" message that the site was blocked right away. Forcing a crawl (crawl all urls) should result in the site getting indexed extremely quickly. Rankings should return to where they were before.
-
The only thing I would add to the existing responses, is that if following a "site:www.mysite.com" query you notice that some key landing pages haven't been indexed then submit them via Webmaster Tools (Fetch as Google).
I would also make sure your sitemap is up to date and submitted via WMT too. It will also tell you how many of the sitemap URLs have been indexed.
These 2 things could speed up your re-indexing. My guess is that if it's a reputable site, and the migration of URLs was done properly, you'll probably get re-indexed quickly anyway.
George
-
Hi Gina,
Yes, that is what I mean. The dev team (or you, if you chose) would get an email that says the robots.txt file had changed. I was inhouse at a non-profit where we had an overseas dev team that wasn't too savvy about SEO, so I was the one who would get the emails, then go and send them an email asking them to fix it.
I don't believe there's a hard and fast answer here, as it in part depends on how quickly your site is crawled.
-
If possible, take a look at the server log files. That should give you a better idea of when/how often Google crawled the site in recent history. The user agent you're looking for is googlebot.
Aside from the robots.txt faux pas, it's also possible that the proper redirects weren't put in place. That would also account for a dip in traffic. Generally WordPress is extensionless. Which means any previous URL that contained an extension won't properly resolve - which means the site would lose a chunk of referral traffic and link equity if the URLs contained an extension (.php, .html, .aspx). Further, if the URL names have been changed from something like /our-non-profit.html to /about-our-non-profit those would require a redirect as well.
I've seen brand new domains index in a matter of days, then rank very well in as little as one month. But that's the exception, not the rule.
Provided proper redirects are in place and nothing too drastic happened to on-page considerations, I would guesstimate two weeks to a month. If you start heading into the month time frame, it's time to look a little deeper.
edit: If the server changed, that would also add another wrinkle to the problem. In the past, one of my lovely hosts decided to force a change on me. It took about a month to recover.
-
Thanks so much for your response KeriMorgret. I'm not sure I fully understand your suggestion unless you are saying that it would have alerted the dev team to the problem? I will pass this on to them and thank you if that is what your intention was.
The developer removed the robot.txt file which fixed the problem and I am trying to ascertain if there is a general expectation on how something like this - a de-indexing - gets reversed within the Google algorithm.
-
I don't know how long it will take for reindexing, but I do have a suggestion (have been in a real similar situation at a non-profit in the past).
Use a monitoring software like https://polepositionweb.com/roi/codemonitor/index.php that will check your robots.txt file daily on your live and any dev servers and email you if there is a change. Also, suggest that the live server's robots.txt file be made read-only, so it's harder to overwrite when updating the site.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does Google encryption of keyword data impact SEO revenue reporting in Google analytics?
Hi there, I know Google has been encrypting SEO keyword data which they rolled out in September 2013. My question is - will this impact SEO revenue figures reported in Google analytics? I have been monitoring SEO revenue figures for a client and they are significantly down even though rankings have not lowered. Is this because of Google's encryption? Could there be another reason? Many thanks!
Reporting & Analytics | | CayenneRed890 -
Google analytcics sub domain dot or not?
Buongiorno from 16 degrees C wetherby UK famous for the Wetherby Whaler Chippy 😉 OK... on this site http://www.philpotts.co.uk/ I've set up sub domain tracking as so:
Reporting & Analytics | | Nightwing
Parent site:
http://www.philpotts.co.uk/ Sub domain
http://shop.philpotts.co.uk/ So my question is please: should a dot be placed in the sub domain line as in : _gaq.push(['_setDomainName', '.philpotts.co.uk']); Some advice places a dot in setDomainName other advice doesnt 😞
Any insights welcome, Grazie,David0 -
What will you do when seeing the rank and traffic are dropping?
No black-hat SEO strategy was adopted, so we suppose it's not a penalty, but ranks and organic traffics are dropping. What will you do in this situation? And how long it takes for you to make sure it's not a normal flux, but something wrong happens on your site? Thanks in advance
Reporting & Analytics | | JonnyGreenwood0 -
Best use of Keyword Rankings report
is there a guide for getting the most out of the keyword ranking tool? I currently have 250 or so keywords, some move up, some move down on a weekly basis but generally trend up. That's about as much info as I gain from it though. anyone got any good tips? thanks
Reporting & Analytics | | benj450 -
Google SEO - Where have I disappeared to?
Okay, so first off Google, I hate you. Before I signed up for SEOMoz, my website was hitting page 9 and page 10 for some ultra difficult keywords. After spending a month using Hubspot and SEOMoz, I finally made it on to page 2 of google, for said 'impossible to rank high' keywords, which I was super happy with. But last week, I login to find that I have disappeared off the top 100 pages of Google for about 100 of my top keywords!!!! What the hell did I do wrong? I tried to please Google, but my website is still indexed, but just not ranked at all for any of my top keywords. The last thing I did before I disappeared overnight was add "follow me" buttons to all my pages and "share this" buttons to all my blogs. Could this be the problem? My website and main keyword is Process Server Is there anyone who could help push me in the right direction? I have no idea what I did wrong. 😕 Martyn
Reporting & Analytics | | spymore0 -
Can't figure this ranking out..
Hi, This is puzzling me. I've been in the second/third position for a week or so for my best keyword. That is for Google US unpersonalized, which is the one that brings more traffic, as far as I understand. It can't get MUCH better. Well, I can be first, but second and third position is really awesome in my case (highly competitive keyword according to SEOMOZ PRO). Then, why on earth my traffic for that keyword was 8 times better a year ago?? I mean, a year ago I received an average of 800 visits per day and now I can barely reach 90 visits per day being in the second / third place. Visits can't increase from 90 to 800 just for increasing one spot. I've never seen in my stats such drop in my rankings. I thought that due to google updates my site was sent below the 20th position or something. But my I was shocked today when I saw that I still have the second/third position Am I crazy or this looks wrong? The page title and description that shows in google hasn't changed, so people looking for that keyword are seeing the same as one year ago. It is not a seasonal or time sensitive keyword. My best guess is that people are now always logged in and results are personalized. Don't know much about personalized results but I don't think you can optimize much for those. If that's the case, then how on earth can we optimize a page if everybody is using personalized results? Is there a way to improve your rankings in those cases? Thanks, Enrique
Reporting & Analytics | | enriquef0 -
Advanced segements Google Analytics
Hi guys, I was to hasty last time marking the given question as answered. So i would like to ask another question on the subject. My original question was how to get a pdf report of 10 specific keywords with information about visits,pages per view. avg, visit time, new visitors and bounce rate. The answer was to create an advanced segment with "or operators" and adding all keywords on an exact match basis. I did this but it still gives me a list of more then 500 keywords in traffic sources organic view. Next to that i get different visitor numbers for some keywords where it says 40 at the normal segments (all visits) but gives me 0 when i turn on the advanced segment. I've been clicking and changing things for a long time until i got almost dizzy. Could someone plz help?
Reporting & Analytics | | 5MMedia0 -
Google Search Results inconsistent from different computers
Recently after some optimization activities - I do not see much movement in search rankings - my client is seeing the results on page 1 position 3 and I see page 2 for the same keyword. How does Google change ranking based on past searches and how can I get an accurate picture of what the actual rank is?
Reporting & Analytics | | devonkrusich0