Thousands of 503 Errors

[osports]

I was just checking Google Webmaster Tools for one of the first times (I know this should have been a regular habit).

I noticed that on Feb 8th we had almost 80K errors of type 503.  This is obviously very alarming because as far as I know our site was up and available that whole day.  This makes me wonder if there is a firewall issue or something else that I'm not aware of.

Any ideas for the best way to determine what's causing this?

Thanks,

Chris

[RobertFisher]

Cyrus,

Thanks for the props, but also cool on the crawl delay link. I wish I could say I knew about it before this answer, but I didn't; cool stuff for bigger high update sites.

Always appreciate what you have to say as I learn a lot from you.

Best

[Cyrus-Shepard]

Hi Chris,

This is a really hard problem to diagnose from the outside like, so I'll just give you my thoughts.

1. Are the URLs throwing the 503 errors real pages? Can they be accessed normally by human visitors through the site? I only mention this because sometimes you get software generating a bunch of random links that go nowhere, and weird stuff starts to happen when Google crawls those URLs. Normally you'd see these result as 404s, however.

2. Is the date in Google Webmaster Tools for the 503 errors recent? Sometimes they log those for a long time after the problem is actually solved, especially for URLs they don't visit much.

3. How often does your site go down?

4. Try performing a "Fetch as Googlebot" test on some of the effected URLs

5. I doubt googlebot is crashing your site, but you could always try a crawl delay

6. If nothing else, you'll find the problem at the serving/hosting level. Can't be much help there, unfortunately.

[osports]

It turns out that the Magento patch did NOT fix the problem.  We are still receiving tens of thousands of 503 errors when Googlebot requests a page.  The site is not down.  I can look in the access_log and see that the request was responded to with a 503 error.

Any ideas?  This has to be killing our chances for organic traffic until this gets resolve.

[osports]

Hi Robert,

Thanks for the response.   It turns out that this is due to a bug in our hosting software, Magento, that results in googlebot not being handled correctly.  Apparently there's a patch that's being tested now.

Thanks,

Chris

[RobertFisher]

Do you know where you are hosted? Have you called them to see if the server is down or intermittently down?

Here is a how to resolve link.

Look at bottom and follow the directions regarding using the wayback machine to see if it is temporary or the server is down for maintenance.

That given, if you give us a url, it is easier to assist you.

Best, let us know.