Longevity of robot.txt files on Google rankings
-
This may be a difficult question to answer without a ton more information, but I'm curious if there's any general thought that could shed some light on the following scenario I've recently heard about and wish to be able to offer some sound advice:
An extremely reputable non-profit site with excellent ranking had gone through a re-design and change-over into WordPress. A robots.txt file was used during development on the dev site on the dev server.
Two months later it was noticed through GA that traffic was way down to the site. It was then discovered that the robot.txt file hadn't been removed and the new site (same content, same nav) went live with it in place. It was removed and a site index forced. How long might it take for the site to re-appear and regain past standing in the SERPs if rankings have been damaged. What would the expected recovery time be?
-
They were paying attention to GA but lapsed and when they checked back in, saw a drop in traffic. Great point about that "critical" message.. The developers did force a crawl and I'm hoping you are correct about the time it might take.
-
Thank you methodicalweb. Great suggestions.
-
Thanks, Travis. You've offered a lot of very interesting points.
I will double-check that they have looked at the server log files, but I'm pretty confident that they have done that.
They did assure me that the proper redirects were done but I'm not sure what they did regarding extensions. There was also a server change.....
-
Thanks for clarifying KeriMorgret. Much appreciated. As are all your thoughts. I will definitely suggest that the monitoring software be used to avoid any future problems. This was such an unnecessary and frustrating experience.
-
If they were paying attention to WMT they would have seen a "critical" message that the site was blocked right away. Forcing a crawl (crawl all urls) should result in the site getting indexed extremely quickly. Rankings should return to where they were before.
-
The only thing I would add to the existing responses, is that if following a "site:www.mysite.com" query you notice that some key landing pages haven't been indexed then submit them via Webmaster Tools (Fetch as Google).
I would also make sure your sitemap is up to date and submitted via WMT too. It will also tell you how many of the sitemap URLs have been indexed.
These 2 things could speed up your re-indexing. My guess is that if it's a reputable site, and the migration of URLs was done properly, you'll probably get re-indexed quickly anyway.
George
-
Hi Gina,
Yes, that is what I mean. The dev team (or you, if you chose) would get an email that says the robots.txt file had changed. I was inhouse at a non-profit where we had an overseas dev team that wasn't too savvy about SEO, so I was the one who would get the emails, then go and send them an email asking them to fix it.
I don't believe there's a hard and fast answer here, as it in part depends on how quickly your site is crawled.
-
If possible, take a look at the server log files. That should give you a better idea of when/how often Google crawled the site in recent history. The user agent you're looking for is googlebot.
Aside from the robots.txt faux pas, it's also possible that the proper redirects weren't put in place. That would also account for a dip in traffic. Generally WordPress is extensionless. Which means any previous URL that contained an extension won't properly resolve - which means the site would lose a chunk of referral traffic and link equity if the URLs contained an extension (.php, .html, .aspx). Further, if the URL names have been changed from something like /our-non-profit.html to /about-our-non-profit those would require a redirect as well.
I've seen brand new domains index in a matter of days, then rank very well in as little as one month. But that's the exception, not the rule.
Provided proper redirects are in place and nothing too drastic happened to on-page considerations, I would guesstimate two weeks to a month. If you start heading into the month time frame, it's time to look a little deeper.
edit: If the server changed, that would also add another wrinkle to the problem. In the past, one of my lovely hosts decided to force a change on me. It took about a month to recover.
-
Thanks so much for your response KeriMorgret. I'm not sure I fully understand your suggestion unless you are saying that it would have alerted the dev team to the problem? I will pass this on to them and thank you if that is what your intention was.
The developer removed the robot.txt file which fixed the problem and I am trying to ascertain if there is a general expectation on how something like this - a de-indexing - gets reversed within the Google algorithm.
-
I don't know how long it will take for reindexing, but I do have a suggestion (have been in a real similar situation at a non-profit in the past).
Use a monitoring software like https://polepositionweb.com/roi/codemonitor/index.php that will check your robots.txt file daily on your live and any dev servers and email you if there is a change. Also, suggest that the live server's robots.txt file be made read-only, so it's harder to overwrite when updating the site.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Which one should I use...Google Analytics or AFS Analytics
Hi There, I am about to implement analytics for a customer site and I am wondering if I should use Google Analytics or https://www.afsanalytics.com ? I am leaning towards Google since it's pretty mainstream but I would love some feedback on the AFS ! Thanks everyone! Gary
Reporting & Analytics | | gdavey0 -
Alexa Rank and Sem Rush Rank
Just wondering if these rankings even matter? One will go up while the other goes down.. and the Alexa ranking doesn't seem to follow any rhyme or reason in relation to the data I see in Google analytics. Anyone else using these tools? Is one a better indicator of a websites health than the other or should we scrap both from our reports?
Reporting & Analytics | | unikey0 -
Bing vs Google Keyword Research
Hi All, I have carried out keyword research based on Google Analytics, Keyword Planner, competitors and various other toolsets.
Reporting & Analytics | | Mark_Ch
My research has been based on performance metrics favored towards Google.
I'm looking to research keyword traffic metrics and suggestions based on Bing Ads Intelligence. My Logic
I have compiled my master list based on Google metrics.
I will carry out Bing search traffic metrics based on my Google master list.
I will carry out Bing keyword suggestions based on my Google master list. My Questions
#1 Will my logic yield me any favorable benefits based on the high number of Google users vs Bing users
#2 Should Bing favor a particular keyword with high search traffic compared to Google, is there a fear that creating a 'seo user friendly website page' could dilute link juice based on the high Google users. Any other useful advice would be welcome. Regards Mark0 -
Best use of Keyword Rankings report
is there a guide for getting the most out of the keyword ranking tool? I currently have 250 or so keywords, some move up, some move down on a weekly basis but generally trend up. That's about as much info as I gain from it though. anyone got any good tips? thanks
Reporting & Analytics | | benj450 -
Setting up goals in Google Analytics
Hello! This question is seems so obvious that I'm also ashamed to ask... almost. If the goal of a website is to have a visitor complete a contact form, when setting up the Goal in Google Analytics (as a "URL destination") the URL of choice would be the form's thank you page, correct? Because that's the page that proves the visitor completed the task we want them the achieve. Right? Thanks!
Reporting & Analytics | | SmileMoreSEO
Erik0 -
Google Analytics underreporting
I have a site selling items, I use paypal buttons and get a report using ipn back from paypal telling me when I have sold a item. I also have a Google analytics event setup on the button, and one on the confirmation page.
Reporting & Analytics | | AlanMosley
For example in the last few days I have sold over 10 items from the website, but have only 2 events from the button and none from the confirmation page. The confirmation page depends on the user getting redirected back from PayPal so I can see that being upper reported, but there is no way you can buy without clicking the button.
I have the chrome addon from google to debug your analytics code and it fires every time.
The figures seem to report only about 20% of events over a long period of time. If it was not working at all, I could assume I have not set things up correctly but this is not the case, I get a regular flow of events but at about 205 of sales.
I am very disappointed with GA, seems a waste of time. anyone else had the same expirence?0 -
Need help setting up a Google analytics goal.
I'm just now getting my feet wet with the goals in GA. I'm trying to figure out how many visitors to a certain page, click on a certain link (which takes them no a certain page on my site.) What's the best way to go about this? Thanks.
Reporting & Analytics | | NoahsDad0 -
Google Analytics help - exporting reports
Hey mozzers, A little help if you will. I am in traffic sources > sources > search > organic I am view keywords and comparing 1 month with another. Now I see what 1 keyword sends 1st month, 2nd month and % Change. I want to export this data to an .xls file. When I do, I end up with a file that shows what 1 keyword sends 1st month and 2nd month, but no % Change! I really need to include this in exported document.... Summary: Analytics hates me. 🙂
Reporting & Analytics | | MirandaP0