Huge increase in server errors and robots.txt

[labelPR]

Hi Moz community!

Wondering if someone can help? One of my clients (online fashion retailer) has been receiving huge increase in server errors (500's and 503's) over the last 6 weeks and it has got to the point where people cannot access the site because of server errors.

The client has recently changed hosting companies to deal with this, and they have just told us they removed the DNS records once the name servers were changed, and they have now fixed this and are waiting for the name servers to propagate again.

These errors also correlate with a huge decrease in pages blocked by robots.txt file, which makes me think someone has perhaps changed this and not told anyone...

Anyone have any ideas here? It would be greatly appreciated! I've been chasing this up with the dev agency and the hosting company for weeks, to no avail.

Massive thanks in advance

[labelPR]

Thank you EGOL, all makes perfect sense and I appreciate your reply. I suspect the problems are mostly centered on the hosting issues, with secondary potential robots.txt issues aswell.

[EGOL]

....it has got to the point where people cannot access the site because of server errors.

As soon as I would see this I would go straight to someone who knows a lot more about servers than I do.  I would start with the host and if I get no help from them within a few hours then I would get someone who knows about servers to dig into this and be ready to quickly move the website to a new host.   If the host does not know how to solve it, and I don't know how to solve it.  Then it is time for bigger guns and possibly a new host - right away.

....they have just told us they moved the DNS records once the name servers were changed, and they have now fixed this and are waiting for the name servers to propagate again.

So, the website is now in the hands of a new host.  It is likely that the problem will be solved here it the old host was the cause of the problem.   Today, DNS propagates quickly,  I am having my morning coffee... if I don't see progress by the time I return from lunch then I am calling a pro.

I think that it is a good idea for anyone who has clients or an important website to have a person or a company that they can call straightaway for a quick couple of hours of investigation or advice.   Two hours of consulting is cheaper than seeing a business throttled for two days.

Also, I have learned to stay away from hosts who offer unlimited bandwidth and similar claims.  When you start to become successful you become unprofitable for them so they either have to limit your resources or confess that their claim of unlimited is an absolute lie.

All of my sites are with hosts who charge me for every bit of resource that I use.  The more I use, the more money they make and when I have a problem they are motivated to get it fixed immediately - because when my biz is dragging they are making less money.  They want me to make money because our interests are in alignment - not the opposite.

Cheap hosts are just as bad as the unlimited guys.  If they have a problem with your website it is cheaper to let you go and lose the few bucks a month that you are paying them than it is to pay their staff to fix things. (But they will not tell you to go to a new host - they will just allow you to have crap service until you decided to move.)     I make sure that the hosts that I use have a number of high profile sites under their care who will not tolerate one minute of BS.   These hosts are not cheap, but I am not interested in cheap, I want reliable.