Recovering from Programmers Error
-
Hey Everybody!
Last year one of my bigger sites hit a snaffu. I was getting about 300k + hits a day from google, and then, when a developper released an update with a robots.txt file that basically blocked google from the entire site.
We didn't notice the bug until a few days later, but by then, it was already too late. My google traffic dropped to 30k a day and I've been having the hardest time coming back ever since.
As a matter of fact, hundreds of sites that were aggregating my content started outranking me for my own terms.
For over a year, I've been working on building what I lost back and everything seemed to be coming together. I was back at 100k+ hits a day
Until today... My developpers repeated the exact same error as last year. They blocked google from crawling my site for over 5 days and now I'm down to 10k se hits a day.
My question : Has anyone encountered this problem before and what did you do to come back?
-
My Friend,
I was having the exactly same problem, and finally my solution was add at operating system level the "Read-Only" attribute for this file.
Hope it help
Claudio
-
I don't have an answer for your traffic, but I've had similar experiences with developers. I ended up using Code Monitor from Pole Position at https://polepositionweb.com/roi/codemonitor/index.php. I had it monitor the contents of the robots.txt file for the live site and all dev sites. Once a day it would check the file for any changes, and email me if there were changes, so I had a max lag time of 24 hours to be notified that the devs had done something again.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Recovering from spam links on MY site
Hey guys, Having a weird situation and wondering if anybody can help. I run a sizable WordPress site with a number of content writers. One of the writer's accounts was hacked and was used to post several dozens of complete spam posts with spun content and links to all sorts of shady sites. Recently the site has begun losing rankings on all sorts of pages. There's no manual penalty or anything, but I'm concerned that we're being penalized for having had these links on the site. Of course, as soon as we found the content, we immediately removed it, reset passwords, etc. But a decent number of the pages were indexed. Does anybody have any experience with this or ideas of what to do about it? Is there somewhere we can talk to Google about it or some way to show that we are not part of bad neighborhoods? Thanks so much for any thoughts, Yon
Intermediate & Advanced SEO | | yon230 -
18,000 'Title Element is too Long' Errors
How detrimental is this in the overall SEO scheme of things? Having checked 3 of our main competitors, they too seem to have similar issues... I am trying to look at a solution but it is proving very difficult! Thanks Andy
Intermediate & Advanced SEO | | TomKing0 -
Penalized and how to recover
Hey guys One of our sites was penalised from a while ago and we use to rank page 1 for our keywords and now w can't be found. Is there a way to recover from this? Also, I came across Rands video about site architecture whilst google were teaching us to generate backlinks, and spammy anchor text what would your suggestion be to recover from this? Any ideas? Here is the site http://free-love-psychic.com/ Any suggestions?
Intermediate & Advanced SEO | | edward-may0 -
Do Structured Data Errors Hurt Your Rankings
Hi, I noticed we had a 161 structured data erros. Almost all of them said "Update missing." I included screenshorts. Are these hurting me? If so, what can I do to fix them? Thanks, Ruben UfzeAFi YjLhimj
Intermediate & Advanced SEO | | KempRugeLawGroup0 -
To recover from Penguin update, shall i remove the links or disavow links?
Hi, One of our websites hit by Penguin update and I now know where the links are coming from. I have chance to remove the links from those incoming links but I am a little confused whether i should just remove the links from incoming links or disavow the links? Thanks
Intermediate & Advanced SEO | | Rubix0 -
Robot.txt error
I currently have this under my robot txt file: User-agent: *
Intermediate & Advanced SEO | | Rubix
Disallow: /authenticated/
Disallow: /css/
Disallow: /images/
Disallow: /js/
Disallow: /PayPal/
Disallow: /Reporting/
Disallow: /RegistrationComplete.aspx WebMatrix 2.0 On webmaster > Health Check > Blocked URL I copy and paste above code then click on Test, everything looks ok but then logout and log back in then I see below code under Blocked URL: User-agent: * Disallow: / WebMatrix 2.0 Currently, Google doesn't index my domain and i don't understand why this happening. Any ideas? Thanks Seda0 -
Recovering from an Automatic penalty
My website dropped from ~15 to 500 + for our main keywords in target markets (but are still doing okay in other countries and for other keywords) I cleaned my site up and contacted Google who told me no manual spam actions were taken on my site. The only thing I can think is that we suffered from an automatic penalty (the drop corresponded with a small panda update). If that is in fact what happened, how do we recover? Also feel free to contribute other ideas about what may have occurred.
Intermediate & Advanced SEO | | theLotter0 -
Google Webmaster Tools Sitemap errors for phantom urls?
Two weeks ago we changed our urls so the correct addresses are all lowercase. Everything else 301 redirects to those. We have submitted and made sure that Google has downloaded our updated sitemap several times since. Even so, Webmaster Tools is reporting 33000 + errors in our sitemap for urls that are no longer in our sitemap and haven't been for weeks. It claims to have found the errors within the last couple of days but the sitemap has been updated for a couple of weeks and has been downloaded by Google at least three times since. Here is our sitemap: http://www.aquinasandmore.com/urllist.xml Here are a couple of urls that Webmaster Tools says are in the sitemap: http://www.aquinasandmore.com/catholic-gifts/Caroline-Gerhardinger-Large-Sterling-Silver-Medal/sku/78664
Intermediate & Advanced SEO | | IanTheScot
Redirect error unavailable
Oct 7, 2011
http://www.aquinasandmore.com/catholic-gifts/Catherine-of-Bologna-Small-Gold-Filled-Medal/sku/78706
Redirect error unavailable
Oct 7, 20110