Severe rank drop due to overwritten robots.txt
-
Hi,
Last week we made a change to drupal core for an update to our website. We accidentally overwrote our good robots.txt that blocked hundreds of pages with the default drupal robots.txt. Several hours after that happened (and we didn't catch the mistake) our rankings dropped from mostly first, second place in Google organic to bottom and mid first page.
Basically I believe we flooded the index with very low quality pages at once and threw a red flag and we got de-ranked.
We have since fixed the robots.txt and have been re-crawled but have not seen a return in rank.
Would this be a safe assumption of what happened? I haven't seen any other sites getting hit in the retail vertical yet in regards to any Panda 2.3 type of update.
Will we see a return in our results anytime soon?
Thanks,
Justin
-
Your present approach is correct. Ensure all these pages are tagged as noindex for now. Remove the block from robots.txt and let Google and Bing crawl these pages.
I would suggest waiting until you are confident all the pages were removed from Google's index, then check Yahoo and Bing. If you decide that robots.txt is the best decision for your company, then you can replace the disallows after confirming your site is no longer affected by these pages.
I would also suggest that, going forward, you ensure any new pages on your site that you do not wish to index always include the appropriate meta tag. If this issue happens again then you will have a layer of protection in place.
-
We're pretty confident thus far that we have flooded the index with about 15,000 low rank URLs all at once. This has happened once in the past a few years back but we didn't flood their index, they were newer pages at the time in which were low quality and could have been seen as spam since there was no real content but adsense so we removed them with a disallow in robots.
We are adding the meta no-index to all of these pages. You're saying we should remove the disallow in robots.txt so googlebot can crawl these pages and see the meta-noindex?
We are a very large site and we're crawled often. We're a PR7 site and MOZrank DA is 79/100. We have dropped from 82.
We're hoping these URLs will be removed quickly, I don't think there is a way of removing 15k links in GWMT without setting off flags also.
-
There is no easy answer for how long it will take.
If your theory about the ranking drop being caused by these pages being added is correct, then as these pages are removed from Google's index, your site should improve. The timeline depends on the size of your site, your site's DA, the PA and links for these particular pages, etc.
If it was my site I would mark the calendar for August 1st to review the issue. I would check all the pages which were mistakenly indexed to be certain they were removed. After, I would check the rankings.
-
Hi Ryan,
Thanks for your response. Actually you are correct. We have found some of the pages that should be no follows still indexed. We are now going to use the noindex, follow meta tags on these pages because we can't afford to have theses pages indexed as they are particularly for clients/users only and are very low quality and have been flagged before.
Now, how long until we see our rank move back? Thats the real big question.
Thanks so much for your help.
Justin
-
That's a great answer Ryan... I wonder, just out of curiosity, if it wouldn't hurt to look at the cached version of the pages if they're indexed? I'd be curious to know if the date they were cached is right near when the robots.txt was changed? I know it wouldn't alter his course of action, but might add further confirmation that this caused the problem?
-
Justin,
Based on the information you provided it's not possible to determine if the robots.txt file was part of the issue. You need to investigate the matter further. Using Google enter a query in an attempt to find some of the previously blocked content. For example, let's assume your site is about SEO but you shared a blog article about your movie review of the latest Harry Potter movie. You may have used robots.txt to block that article because it is unrelated to your site's focus. Perform a search for "Harry Potter insite:mysite.com" replacing mysite.com with your main web address. If the search returns your article, then you know the content was indexed. Try this approach for several of your previously blocked areas of the website.
If you find this content in SERPs, then you need to have it removed. The best thing to do is add the "noindex, follow" tags to all these pages, then remove the block from your robots.txt file.
The problem is that with the block in place on your robots.txt file, Google cannot see the new meta tag and does not know to remove the content from it's index.
One last item to mention. Google does have a URL removal tool but that would not be appropriate in this instance. That tool is designed to remove a page which causes direct damage by being in the index. Trade secrets or other confidential information can be removed with this tool.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Disallow wildcard match in Robots.txt
This is in my robots.txt file, does anyone know what this is supposed to accomplish, it doesn't appear to be blocking URLs with question marks Disallow: /?crawler=1
Technical SEO | | AmandaBridge
Disallow: /?mobile=1 Thank you0 -
Multiple robots.txt files on server
Hi! I have previously hired a developer to put up my site and noticed afterwards that he did not know much about SEO. This lead me to starting to learn myself and applying some changes step by step. One of the things I am currently doing is inserting sitemap reference in robots.txt file (which was not there before). But just now when I wanted to upload the file via FTP to my server I found multiple ones - in different sizes - and I dont know what to do with them? Can I remove them? I have downloaded and opened them and they seem to be 2 textfiles and 2 dupplicates. Names: robots.txt (original dupplicate)
Technical SEO | | mjukhud
robots.txt-Original (original)
robots.txt-NEW (other content)
robots.txt-Working (other content dupplicate) Would really appreciate help and expertise suggestions. Thanks!0 -
Terrible branded ranking
I am at lost. My website has been live for about 3 months. Typing my name or my direct page title into google puts me on page 15. My site is indexed by google, a site:mywebsitename search - produces 202 results. My on page SEO isn't amazing buy it isn't terrible either. The website has 35+, 500 words+ reviews and a few incoming links from some occasional link building. 0 duplicate content. The website is animexcess.com
Technical SEO | | WebsiteEditor
My thoughts on what might be causing the problem, i am hoping that someone with a bit more SEO knowledge than myself can point me in the right direction. During development, i had duplicate content up on the site as placeholder while i wrote my own - All duplicate content has been removed for about 3 months now, but can this still be affecting me ranking? and to this extent? The entire website is in a sub-directory with 301 redirects - can this be affecting my ranking to this extent? The domain had a prior owner. Waybackmachine tells me that the domain had a page up in 2007. I bought it in 2010. If the prior owner built up a bad rep on this domain can is still be the cause for poor ranking 6 years later? I cannot file a reconsideration request since there are no manual actions taken against my site in WMT, any way around this? Its only been three months. I know 3 months isn't a very long time in the google ranking world but my my 2 week old, built in 1 week, secondary website is ranking on page 1 for some pretty solid terms, so i don't think 3 months is to short a time to expect a bit of results. My website is being crawled daily.
Any advice on how to move forward with fixing my website ranking will be much obliged. Thank you.0 -
Robots.txt checker
Google seems to have discontinued their robots.txt checker. Is there another tool that I can use to check my text instead? Thanks!
Technical SEO | | theLotter0 -
Confirming Robots.txt code deep Directories
Just want to make sure I understand exactly what I am doing If I place this in my Robots.txt Disallow: /root/this/that By doing this I want to make sure that I am ONLY blocking the directory /that/ and anything in front of that. I want to make sure that /root/this/ still stays in the index, its just the that directory I want gone. Am I correct in understanding this?
Technical SEO | | cbielich0 -
Sudden Drop in Keyword Rankings
We launched http://www.manufacturedfun.com/ earlier this year and had been ranking 1 & 2 on Google SERPs for the keywords we optimized, but last week we experienced a sudden drop in rankings that pretty much took us off the radar. For instance, 'popcorn machines' went as far back as page 5 and 'popcorn poppers' dropped even further to page 9. We are currently working on fixing the numerous Duplicate Page Title and Duplicate Page Content errors identified by SEOmoz, but since we have had those for about 6 months and ranked well anyway, I wonder if there's something else that we are missing. Any insight you can offer will be sincerely appreciated. Thank you!
Technical SEO | | GRIP-SEO0 -
Panda and unnatural links caused ranking drop
Hi I have been approached to do some SEO work for a site that has been hit badly by the latest panda update 3.3, they have also had a warning in their Google webmaster tools account saying they had unnatural looking links to their site, they received this in 26 Feb and that prompted them to stop working with their excising seo company and look for a new one. Apparently their rankings for the keywords they were targeting have dropped dramatically, but it looks like just those they were actively building back links for, other phrases do not look affected. Before I take them on I want to be clear that it is possible to help them reclaim their rankings? I have checked the site and the on-page seo is good, the site build is good, just a few errors to fix but the links that have been built by the seo company are low quality with a lot of spun articles and the same anchor text so I see what the Google webmaster tools message is refuring to. I do not think these links can be removed as there is no contact details on the sites I checked I have not checked all of them but a random sample does not show promise, they are from low authority domains. So if I am to take them on as a client and help them to regain their previous rankings what is the best strategy? Obviously they want results yesterday and from our phone call they would rather someone else did the work than them, so my initial response of add some better quality content that others in your industry would link to as a reference did not go down well, to be fair I think it is a time issue there are only 3 people in the company and they are not technical at all. Thanks for your help Sean
Technical SEO | | ske110 -
How long does it take for traffic to bounce back from and accidental robots.txt disallow of root?
We accidentally uploaded a robots.txt disallow root for all agents last Tuesday and did not catch the error until yesterday.. so 6 days total of exposure. Organic traffic is down 20%. Google has since indexed the correct version of the robots.txt file. However, we're still seeing awful titles/descriptions in the SERPs and traffic is not coming back. GWT shows that not many pages were actually removed from the index but we're still seeing drastic rankings decreases. Anyone been through this? Any sort of timeline for a recovery? Much appreciated!
Technical SEO | | bheard0