Severe rank drop due to overwritten robots.txt
-
Hi,
Last week we made a change to drupal core for an update to our website. We accidentally overwrote our good robots.txt that blocked hundreds of pages with the default drupal robots.txt. Several hours after that happened (and we didn't catch the mistake) our rankings dropped from mostly first, second place in Google organic to bottom and mid first page.
Basically I believe we flooded the index with very low quality pages at once and threw a red flag and we got de-ranked.
We have since fixed the robots.txt and have been re-crawled but have not seen a return in rank.
Would this be a safe assumption of what happened? I haven't seen any other sites getting hit in the retail vertical yet in regards to any Panda 2.3 type of update.
Will we see a return in our results anytime soon?
Thanks,
Justin
-
Your present approach is correct. Ensure all these pages are tagged as noindex for now. Remove the block from robots.txt and let Google and Bing crawl these pages.
I would suggest waiting until you are confident all the pages were removed from Google's index, then check Yahoo and Bing. If you decide that robots.txt is the best decision for your company, then you can replace the disallows after confirming your site is no longer affected by these pages.
I would also suggest that, going forward, you ensure any new pages on your site that you do not wish to index always include the appropriate meta tag. If this issue happens again then you will have a layer of protection in place.
-
We're pretty confident thus far that we have flooded the index with about 15,000 low rank URLs all at once. This has happened once in the past a few years back but we didn't flood their index, they were newer pages at the time in which were low quality and could have been seen as spam since there was no real content but adsense so we removed them with a disallow in robots.
We are adding the meta no-index to all of these pages. You're saying we should remove the disallow in robots.txt so googlebot can crawl these pages and see the meta-noindex?
We are a very large site and we're crawled often. We're a PR7 site and MOZrank DA is 79/100. We have dropped from 82.
We're hoping these URLs will be removed quickly, I don't think there is a way of removing 15k links in GWMT without setting off flags also.
-
There is no easy answer for how long it will take.
If your theory about the ranking drop being caused by these pages being added is correct, then as these pages are removed from Google's index, your site should improve. The timeline depends on the size of your site, your site's DA, the PA and links for these particular pages, etc.
If it was my site I would mark the calendar for August 1st to review the issue. I would check all the pages which were mistakenly indexed to be certain they were removed. After, I would check the rankings.
-
Hi Ryan,
Thanks for your response. Actually you are correct. We have found some of the pages that should be no follows still indexed. We are now going to use the noindex, follow meta tags on these pages because we can't afford to have theses pages indexed as they are particularly for clients/users only and are very low quality and have been flagged before.
Now, how long until we see our rank move back? Thats the real big question.
Thanks so much for your help.
Justin
-
That's a great answer Ryan... I wonder, just out of curiosity, if it wouldn't hurt to look at the cached version of the pages if they're indexed? I'd be curious to know if the date they were cached is right near when the robots.txt was changed? I know it wouldn't alter his course of action, but might add further confirmation that this caused the problem?
-
Justin,
Based on the information you provided it's not possible to determine if the robots.txt file was part of the issue. You need to investigate the matter further. Using Google enter a query in an attempt to find some of the previously blocked content. For example, let's assume your site is about SEO but you shared a blog article about your movie review of the latest Harry Potter movie. You may have used robots.txt to block that article because it is unrelated to your site's focus. Perform a search for "Harry Potter insite:mysite.com" replacing mysite.com with your main web address. If the search returns your article, then you know the content was indexed. Try this approach for several of your previously blocked areas of the website.
If you find this content in SERPs, then you need to have it removed. The best thing to do is add the "noindex, follow" tags to all these pages, then remove the block from your robots.txt file.
The problem is that with the block in place on your robots.txt file, Google cannot see the new meta tag and does not know to remove the content from it's index.
One last item to mention. Google does have a URL removal tool but that would not be appropriate in this instance. That tool is designed to remove a page which causes direct damage by being in the index. Trade secrets or other confidential information can be removed with this tool.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Rankings Dropped to Nothing
We're kind of in crisis mode, as our ad revenue is about to take a huge hit. Hoping someone can help me figure out what to do next. Site: https://indoorgardening.com Here's what I did (below) that I think broke things somehow. I'm clearly not an SEO expert but thought I was making things better. And things did improve over the last week or so but then fell apart 2 days ago. 1. Most posts did not have a Yoast focus keyword. I added keyword phrases and used Yoast suggestions to optimize for that and for readability. 2. In some cases I changed post titles to better reflect the keyword phrase. 3. In some cases I changed the slug per Yoast's suggestion and did a 301 redirect from the old slug to the new one. 4. I used Grammarly to fix all spelling, grammar, punctuation, etc. 5. Some Yoast-suggested changes that were made: Image alt tags, subheading structures, adding keyword to subheadings, first paragraph, and meta description, changed sentence length for those over 20 words to clean up the text, added transition words where applicable, reworded passive voice sentences, added internal links when needed, eliminated consecutive sentences (first word), improved Flesch reading ease when necessary. 6. I also added or changed Amazon affiliate links where needed and swapped out images when necessary. Results:
Technical SEO | | Jbyron
I started this project about 3 weeks ago. On 11/29 we had one of our highest traffic days, with 1017 hits coming from Google. On 11/30, 257 hits came from Google, and on 12/1, 3 (three!!) hits came from Google. At this point, 82 of 89 posts have a double green "Good" score in Yoast; 6 are "OK" and 1 does not have a focus keyword designated. Thanks in advance for any help anyone can provide. -John0 -
Sudden Drop in Rankings
I manage a site and this past month have noticed some pretty significant drops in rankings for some of our main keywords. I'm trying to understand why and what I can do to correct it. We did do some 301 redirects a little over a month ago so I've been wondering if that has anything to do with it as Google indexes the new URLs. One thing I did notice in Moz is that it is showing that Google is indexing the new URLs but associating them with the wrong keywords in the wrong location. The company services three locations and each location has a service page on the site. But Moz is showing that in Atlanta, the indexed URL is the Nashville one, etc. Is there any way to affect that? Another thing to note is that organic search traffic is actually up so it doesn't appear to be affecting them which I find a bit strange. Just looking for any insight as to things I should be looking into to explain the sudden drops. Thanks!
Technical SEO | | maghanlinchpinsales0 -
Is there a limit to how many URLs you can put in a robots.txt file?
We have a site that has way too many urls caused by our crawlable faceted navigation. We are trying to purge 90% of our urls from the indexes. We put no index tags on the url combinations that we do no want indexed anymore, but it is taking google way too long to find the no index tags. Meanwhile we are getting hit with excessive url warnings and have been it by Panda. Would it help speed the process of purging urls if we added the urls to the robots.txt file? Could this cause any issues for us? Could it have the opposite effect and block the crawler from finding the urls, but not purge them from the index? The list could be in excess of 100MM urls.
Technical SEO | | kcb81780 -
Combine two websites or keep them separate after Penguin 3 ranking drop and gain
Since 1995 we had one website which combined our wedding and portrait photography business.(website A) Three years ago we created a new website and new domain name for the portrait photography side. (website B) We did not delete the portrait information from website A. Both sites were ranking well on page one. After Penguin 3 website B is no longer ranking at all. Website A is still ranking well and one of the original portrait pages on the website A is now ranking on page one also. I am wondering what to do and considering the below options: 1. Should I go back to a combined wedding and portrait site? (and delete website B) 2. Should I create a "301" from the original portrait page on website A that is now ranking, to website B that is no longer ranking, and delete all portrait content from website A? 3. Will having a combined wedding and portrait site be harder to rank as they are competing against each other, or will they help each other rank? Any comments or advice greatly appreciated. Thanks
Technical SEO | | annaberg0 -
Image Impression Drop
On August 8th we started to see our Image Impressions in Google Webmaster Tools start to plummet. Has anyone else run into this issue? I have not been able to find any news on Google Algos which my have caused this. Any other ideas what could have caused this? jxPmLwO
Technical SEO | | joebuilder0 -
Google Impressions Drop Due to Expired SSL
Recently I noticed a huge drop in our clients Google Impressions via GWMT from 900 impressions to 70 overnight on October 30, 2012 and has remained this way for the entire month of November 2012. The SSL Cert had expired in mid October due to the notification message for renewal going to the SPAM folder and being missed. Is it possible for an SSL expiry to be related to this massive drop in daily impressions which in-turn has also effected traffic? I also can't see any evidence of duplicate pages (ie. https and http) being indexed but to be honest I'm not the one doing the SEO therefore haven't been tracking this. Thanks for your help! Chris
Technical SEO | | MeMediaSEO0 -
Malware ranking drops
Hi, One of our sites got hit with malware in November. We cleaned it up and sent a reconsideration request and no malware warnings are in WMT anymore. We still haven't seen an improvement in rankings even though people say it can take up to 90days? Organic traffic is down 40% and still dropping. Any advice? Thanks
Technical SEO | | Sayers0 -
Robots.txt
Hi there, My question relates to the robots.txt file. This statement: /*/trackback Would this block domain.com/trackback and domain.com/fred/trackback ? Peter
Technical SEO | | PeterM220