Huge increase in server errors and robots.txt
-
Hi Moz community!
Wondering if someone can help? One of my clients (online fashion retailer) has been receiving huge increase in server errors (500's and 503's) over the last 6 weeks and it has got to the point where people cannot access the site because of server errors.
The client has recently changed hosting companies to deal with this, and they have just told us they removed the DNS records once the name servers were changed, and they have now fixed this and are waiting for the name servers to propagate again.
These errors also correlate with a huge decrease in pages blocked by robots.txt file, which makes me think someone has perhaps changed this and not told anyone...
Anyone have any ideas here? It would be greatly appreciated!
I've been chasing this up with the dev agency and the hosting company for weeks, to no avail.
Massive thanks in advance
-
Thank you EGOL, all makes perfect sense and I appreciate your reply. I suspect the problems are mostly centered on the hosting issues, with secondary potential robots.txt issues aswell.
-
....it has got to the point where people cannot access the site because of server errors.
As soon as I would see this I would go straight to someone who knows a lot more about servers than I do. I would start with the host and if I get no help from them within a few hours then I would get someone who knows about servers to dig into this and be ready to quickly move the website to a new host. If the host does not know how to solve it, and I don't know how to solve it. Then it is time for bigger guns and possibly a new host - right away.
....they have just told us they moved the DNS records once the name servers were changed, and they have now fixed this and are waiting for the name servers to propagate again.
So, the website is now in the hands of a new host. It is likely that the problem will be solved here it the old host was the cause of the problem. Today, DNS propagates quickly, I am having my morning coffee... if I don't see progress by the time I return from lunch then I am calling a pro.
I think that it is a good idea for anyone who has clients or an important website to have a person or a company that they can call straightaway for a quick couple of hours of investigation or advice. Two hours of consulting is cheaper than seeing a business throttled for two days.
Also, I have learned to stay away from hosts who offer unlimited bandwidth and similar claims. When you start to become successful you become unprofitable for them so they either have to limit your resources or confess that their claim of unlimited is an absolute lie.
All of my sites are with hosts who charge me for every bit of resource that I use. The more I use, the more money they make and when I have a problem they are motivated to get it fixed immediately - because when my biz is dragging they are making less money. They want me to make money because our interests are in alignment - not the opposite.
Cheap hosts are just as bad as the unlimited guys. If they have a problem with your website it is cheaper to let you go and lose the few bucks a month that you are paying them than it is to pay their staff to fix things. (But they will not tell you to go to a new host - they will just allow you to have crap service until you decided to move.) I make sure that the hosts that I use have a number of high profile sites under their care who will not tolerate one minute of BS. These hosts are not cheap, but I am not interested in cheap, I want reliable.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Block session id URLs with robots.txt
Hi, I would like to block all URLs with the parameter '?filter=' from being crawled by including them in the robots.txt. Which directive should I use: User-agent: *
Intermediate & Advanced SEO | | Mat_C
Disallow: ?filter= or User-agent: *
Disallow: /?filter= In other words, is the forward slash in the beginning of the disallow directive necessary? Thanks!1 -
How to Increase PR? I am using joomla platform for my Website.
Hi There, i have question how to increase the Pagerank? My website health according ahrefs is 90%. but still no pagerank. please help.
Intermediate & Advanced SEO | | GarySahota0 -
Can I use the old website content on the new website, after deleting it from the server?
My website nowwhatstudio.com hit by google pure spam and google applied manual spam action to the website. I create new website (nowwhatmoments.com) with the same content from the old spam action website (nowwhatstudio.com). As google removed my old website content from search indexed. Can I use the same content for a new website? If I delete my old website from the server, after that Can I use the old website content for a new website? Or Can make edits the old website content and make it 80% original for a new website?
Intermediate & Advanced SEO | | bondhoward0 -
Cannot Increase Ranking For a Keyword Phrase
I've been working on the keyword phrase, "Niceville Assisted Living" for the website: http://nicevilleassistedliving.com and my increase in rankings has pretty much stalled. When I first started working on this website, a lot of the content was duplicated (which we took care of by plugging in unique content), there were locations listed on the homepage that were throwing my rankings off, I've created blog posts each week (we've even tried posting one post every day for a week), added the Facebook feed to the homepage, corrected errors in the theme, and I'm trying to get a resources page built. I know content is a very, very large part of SEO.. but it seems like the content I am plugging in isn't helping. There aren't any errors in Webmaster Tools and my keyword density is fairly close to the website ranking #1. I think my biggest problem is backlinks. Other websites have quite a few whereas the website I'm working on doesn't have any (I'm working on that, but the number I have doesn't compare to the websites ranking in the top three). I'm stumped as to what to do next. Does anyone have suggestions to improve the ranking for this keyword phrase?
Intermediate & Advanced SEO | | ReviveMedia0 -
Can't find X-Robots tag!
Hi all. I've been checking out http://www.unthankbooks.com/ as it seems to have some indexing problems. I ran a server header check, and got a 200 response. However, it also shows the following: X-Robots-Tag:
Intermediate & Advanced SEO | | Blink-SEO
noindex, nofollow It's not in the page HTML though. Could it be being picked up from somewhere else?0 -
How important are sitemap errors?
If there aren't any crawling / indexing issues with your site, how important do thing sitemap errors are? Do you work to always fix all errors? I know here: http://www.seomoz.org/blog/bings-duane-forrester-on-webmaster-tools-metrics-and-sitemap-quality-thresholds Duane Forrester mentions that sites with many 302's 301's will be punished--does any one know Googe's take on this?
Intermediate & Advanced SEO | | nicole.healthline0 -
Should I Host New Blog On Different Server?
I am planning on starting a new blog focused on a very specific subject that is related to my current Lead Generation website. I am doing this because I want to build rankings for this site and have it link back to my Lead Generation website to increase its rankings (this is the main purpose). My question is do I need to host it on another server so Google will not see it as having the same IP or Subnet? Note: I already have one blog linked to my Lead Generation website under the same domain. This new blog would be linked to an entirely different domain.
Intermediate & Advanced SEO | | MyNet0