Longevity of robot.txt files on Google rankings
-
This may be a difficult question to answer without a ton more information, but I'm curious if there's any general thought that could shed some light on the following scenario I've recently heard about and wish to be able to offer some sound advice:
An extremely reputable non-profit site with excellent ranking had gone through a re-design and change-over into WordPress. A robots.txt file was used during development on the dev site on the dev server.
Two months later it was noticed through GA that traffic was way down to the site. It was then discovered that the robot.txt file hadn't been removed and the new site (same content, same nav) went live with it in place. It was removed and a site index forced. How long might it take for the site to re-appear and regain past standing in the SERPs if rankings have been damaged. What would the expected recovery time be?
-
They were paying attention to GA but lapsed and when they checked back in, saw a drop in traffic. Great point about that "critical" message.. The developers did force a crawl and I'm hoping you are correct about the time it might take.
-
Thank you methodicalweb. Great suggestions.
-
Thanks, Travis. You've offered a lot of very interesting points.
I will double-check that they have looked at the server log files, but I'm pretty confident that they have done that.
They did assure me that the proper redirects were done but I'm not sure what they did regarding extensions. There was also a server change.....
-
Thanks for clarifying KeriMorgret. Much appreciated. As are all your thoughts. I will definitely suggest that the monitoring software be used to avoid any future problems. This was such an unnecessary and frustrating experience.
-
If they were paying attention to WMT they would have seen a "critical" message that the site was blocked right away. Forcing a crawl (crawl all urls) should result in the site getting indexed extremely quickly. Rankings should return to where they were before.
-
The only thing I would add to the existing responses, is that if following a "site:www.mysite.com" query you notice that some key landing pages haven't been indexed then submit them via Webmaster Tools (Fetch as Google).
I would also make sure your sitemap is up to date and submitted via WMT too. It will also tell you how many of the sitemap URLs have been indexed.
These 2 things could speed up your re-indexing. My guess is that if it's a reputable site, and the migration of URLs was done properly, you'll probably get re-indexed quickly anyway.
George
-
Hi Gina,
Yes, that is what I mean. The dev team (or you, if you chose) would get an email that says the robots.txt file had changed. I was inhouse at a non-profit where we had an overseas dev team that wasn't too savvy about SEO, so I was the one who would get the emails, then go and send them an email asking them to fix it.
I don't believe there's a hard and fast answer here, as it in part depends on how quickly your site is crawled.
-
If possible, take a look at the server log files. That should give you a better idea of when/how often Google crawled the site in recent history. The user agent you're looking for is googlebot.
Aside from the robots.txt faux pas, it's also possible that the proper redirects weren't put in place. That would also account for a dip in traffic. Generally WordPress is extensionless. Which means any previous URL that contained an extension won't properly resolve - which means the site would lose a chunk of referral traffic and link equity if the URLs contained an extension (.php, .html, .aspx). Further, if the URL names have been changed from something like /our-non-profit.html to /about-our-non-profit those would require a redirect as well.
I've seen brand new domains index in a matter of days, then rank very well in as little as one month. But that's the exception, not the rule.
Provided proper redirects are in place and nothing too drastic happened to on-page considerations, I would guesstimate two weeks to a month. If you start heading into the month time frame, it's time to look a little deeper.
edit: If the server changed, that would also add another wrinkle to the problem. In the past, one of my lovely hosts decided to force a change on me. It took about a month to recover.
-
Thanks so much for your response KeriMorgret. I'm not sure I fully understand your suggestion unless you are saying that it would have alerted the dev team to the problem? I will pass this on to them and thank you if that is what your intention was.
The developer removed the robot.txt file which fixed the problem and I am trying to ascertain if there is a general expectation on how something like this - a de-indexing - gets reversed within the Google algorithm.
-
I don't know how long it will take for reindexing, but I do have a suggestion (have been in a real similar situation at a non-profit in the past).
Use a monitoring software like https://polepositionweb.com/roi/codemonitor/index.php that will check your robots.txt file daily on your live and any dev servers and email you if there is a change. Also, suggest that the live server's robots.txt file be made read-only, so it's harder to overwrite when updating the site.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Analytics / Facebook UTM
Hello, I have a quick question. I am setting up conversion tracking for my Facebook ads, so I am giving each ad set a tracking URL (UTM) in order to see which ads are converting etc. Is it possible to see on analytics how much I am spending on these ads or a cost per conversion? Or does the tracking merely track general analytics data such as bounce rate, exit rate, revenue generated etc? Kind Regards, James
Reporting & Analytics | | SO_UK0 -
Google Analytics Average Position
I'm looking at Google Analytics -> Acquisition -> Search Engine Optimization -> Queries reports. I'm looking at keywords and the average position. What Google reports and what I see in a Google incognito search is different (usually my search is much lower). For example, for one search term, Google reports 5.8 average position and every time I search it is 8. My local result is 4. Anyone know why this is? I'm wondering if Google is averaging the Local results into number?
Reporting & Analytics | | CalicoKitty20000 -
Google Analytics complexe solution?
Hello, We have Google Analytics on our website and we have started to track the conversions.
Reporting & Analytics | | lunacloud
Basically we have a goal with 3 steps: Account Details (Personal Information) Confirmation (Mobile Confirmation Code) Email ( confirmation link) On the last step (Destination Goal) we send an email to the customer with the account confirmation link, the tracking works perfectly. Our problem is with the Goal Completions on "Traffic Sources" >> "AdWords" >> "Campaigns", Analytics doesn't add the conversions. This problem is related to the email confirmation? There is any solution to overcome this problem? Thank you! yzujFZU.png f4fay1G.png0 -
Page Rank - logarithmic or exponential
Possibly a really stupid question. Is Page Rank logarithmic or exponential? I've seen a lot of people talking about Page Rank saying it's logarithmic but when they describe it they're actually talking about an exponential scale. (Apologies if I'm showing a basic misunderstanding in mathematical knowledge - I studied Drama)
Reporting & Analytics | | BenFox0 -
Google Analytics set up for non-canonicalized domains
Our client's website is non-canonicalized (www.example.com & example.com load the same thing). Google seems to have made a preference for the www, but canonicalizing to www breaks their Flash website. All we're really trying to do at this time is install Google Analytics for them. What's the smartest way to make sure that both www.example.com and example.com are treated exactly the same by Google Analytics? Google Developers: Domains & Directories states that by default visit data will be separately collected between the two domains, although I found no references to the common www/naked domain issue. In stackoverflow: Does google analytics combine naked domains with the www subdomain? Török Gábor says, "Yes, users will be tracked, but the same visitor coming from www.datalookups.com and datalookups.com will be counted as two different visitors." On the same page, Open SEO says, "This is completely false: www.domain.tld and domain.tld are treaded just the same, and get the same value for the HASH code (the number at the start of each __utm cookie). This an exception: every other subdomain.domain.tld will be handeld as a distinct web site". Can any Analytics experts help me sort this out? Thanks!
Reporting & Analytics | | GOODSIR0 -
Setting up Google Analytics for Subsites
I currently have one main .com site and am planning on launching geo-location subsites .co.uk, .com.au, .ru, etc... Traffic will flow between both sites and some of the content on the subsites will be duplicate and therefore include a canonical tag to the main site. I want to set up GA to capture who is going to the subsites and vice versa and correctly capture crossover traffic. Any advice on implementing advanced analytics directly (or links to sources that will direct me the right direction for this project)
Reporting & Analytics | | theLotter0 -
Does Google Analytics parse visits from search apps?
Does anyone know if Google Analytics reports visits to your website differently from individual search apps like Google and Bing? Or do they just treat them the same as any other keyword visit from Google or Bing search engine? I suppose the end result is probably the same as in the Google app you're using Google so it would just be a different access point versus a new tool. I'm just curious if there is a way to see how many Mobile visits are coming from the apps vs the browser. For me personally I have the Google and Bing iOS apps installed but rarely use them, opting for the Safari search bar 99% of the time.
Reporting & Analytics | | nsauser0 -
Do Google penalise you for having too many 404's?
Hi There I have been doing some work reducing the number of 404's displayed in the Crawl Errors found in Googles Webmaster Tools. We had a lot of products that were no longer available so have now been removed to reduce the number of 404s that had been found. However, there are a number of URLs that have been crawled that do not exist on our website and have been flagged in the list of Crawl Errors. I want to know if Google will penalise us for this, perhaps affecting our quality score or if they can see that this is something out of our control. This site for example: http://sibd.com/com_offers_unique_gifts.html has generated a lot of truncated URLs on its site that link to pages that don't exist on our site: http://www.arenaflowers.com/flowers/pri… That is the exact link that it is trying to locate. Here is the report for that particular link. As you can see the content has been scraped by other sites which has spread the problem further. Pages that link to http://www.arenaflowers.com/flowers/pri.. URL Discovery Date
Reporting & Analytics | | ArenaFlowers.com
http://www.justsearchit.com.au/for_flowers_offers,3.html
Sep 12, 2011
http://sibd.com/offers_unique_gifts_for.html
Sep 11, 2011
http://sibd.com/offers_unique_gifts.html
Sep 11, 2011
http://sibd.com/com_offers_unique_gifts_for.html
Sep 11, 2011
http://www.flexfinder.com/flowers_offers_unique_gifts.html
Sep 10, 2011
http://sibd.com/offers_unique_gifts_with.html
Sep 10, 2011
http://sibd.com/com_offers_unique_gifts.html
Sep 10, 2011
http://sibd.com/of_flowers_offers_from.html
Sep 9, 2011
http://arama.frmpc.com/for_flowers_for_less_ltd_includes.html
Sep 9, 2011
http://arama.frmpc.com/flowers_for_less_than_do.html
Sep 9, 2011 I have spotted a lot of these and currently have around 3.3K 404s in total, a majority are from sites we don't control. Is there an acceptable number of 404s a site should aim for and is the above something we should address or are Google smart enough to work out that we can't fix this ourselves? Thanks! Sam.0