Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Can an incorrect 301 redirect or .htaccess code cause 500 errors?
-
Google Webmaster Tools is showing the following message:
_Googlebot couldn't access the contents of this URL because the server had an internal error when trying to process the request. These errors tend to be with the server itself, not with the request. _
Before I contact the person who manages the server and hosting (essentially asking if the error is on his end) is there a chance I could have created an issue with an incorrect 301 redirect or other code added to .htaccess incorrectly?
Here is the 301 redirect code I am using in .htaccess:
RewriteEngine On
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/.]+/)*(index.html|default.asp)\ HTTP/
RewriteRule ^(([^/.]+/)*)(index|default) http://www.example.com/$1 [R=301,L]
RewriteCond %{HTTP_HOST} !^(www.example.com)?$ [NC]
RewriteRule (.*) http://www.example.com/$1 [R=301,L]
Could adding the following code after that in the .htaccess potentially cause any issues?
BEGIN EXPIRES
<ifmodule mod_expires.c="">ExpiresActive On
ExpiresDefault "access plus 10 days"
ExpiresByType text/css "access plus 1 week"
ExpiresByType text/plain "access plus 1 month"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
ExpiresByType application/x-javascript "access plus 1 month"
ExpiresByType application/javascript "access plus 1 week"
ExpiresByType application/x-icon "access plus 1 year"</ifmodule>END EXPIRES
(Edit) I'd like to add that there is a Wordpress blog on the site too at www.example.com/blog with the following code in it's .htaccess:
BEGIN WordPress
<ifmodule mod_rewrite.c="">RewriteEngine On
RewriteBase /blog/
RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /blog/index.php [L]</ifmodule>END WordPress
Thanks
-
Just to follow up on your last question about 404s, Kim...
No, having a bunch of 404s like that will be no more work for the server than if they were landing on actual blog pages - in fact somewhat less work as the 404 page generally has less content and far fewer database calls.
Also, a page timing out due to server load (server working too hard) doesn't generally result in a 500 error, it just returns a timed-out error. 500 errors are delivered when something actually breaks the server's ability to deliver the correct page content.
Paul
-
Wow, you are very quickly and easily making me much better at what I do:) Thanks for that.
I actually just updated the code a couple days ago by adding the Expires code and fixing the redirect. Maybe the previous double 301 redirect could be the culprit? Or - something I mentioned in another question - there were a ton of 404s because of a blog that wasn't redirected to the /blog subdirectory correctly, which I fixed recently. Could something like that cause the server to work to hard and return a 500 server error?
I'll definitely check out the logs and Pingdom.
Great information and advice.
-
Sorry - and to be clear about your htaccess testing question - no there's no "tool" I've ever heard of. You test it by doing exactly as you've done - ensuring that pages respond correctly and with correct headers. Then you implement a monitoring system to ensure that you know every time that correct behaviour fails. That way you can get the site back up quickly, and have a record of when & how often it happened so you can properly troubleshoot if you have an issue.
Three troubleshooting steps
- become aware as soon as there is a problem
- fix the problem asap to minimize impact on users
- investigate and fix the root cause so it doesn't happen again.
All of these steps depend on a monitoring/alerting system, otherwise you'll always be behind the curve and/or working in the dark.
Hope that helps?
Paul
-
Great answer Paul.
-
As far as I understand, Kimberly, you've only changed the htaccess in the last day or 2? in which case the server error would have been from before your updates.
As far as monitoring - you can check the server error logs (via FTP or in cPanel if that's what the hosting account uses) to check for frequent 500-level server errors.
In addition, I strongly recommend that all commercial sites must have uptime monitoring in place. I like to use Pingdom's paid basic plan which allows monitoring of up to 10 pages. I then select a number of relevant pages and set the tool to test each page, and to check for an actual text snippet on each page (using their custom settings). I monitor things like the home page, the blog home page, a blog post, a blog category page, and critical call-to-action pages. Basically different types of page templates that might respond differently to server issues. plus critical money-making pages.
This way, Pingdom will alert you immediately any time those pages don't respond normally (like when a server gives back a 500 error, or the server goes unresponsive due to overload etc). Monitoring these pages every minute is the ONLY way to really know whether your server and website software are performing properly and consistently. This is a critical component of any professionally run website, in my opinion.
Often Pingdom confirms that things are running fine, but I literally can't count the number of times I've instituting uptime monitoring for new clients, only to find the site has huge downtime no one was really aware of, because they just aren't on their own site often enough to know when it's down. (And you certainly shouldn't be relying on customers to inform you the site has issues. By then it's FAR too late.)
Paul
P.S There are certainly other uptime monitoring systems out there, some are even free. I recommend Pingdom because I've used it for years and it's been consistently excellent. Also, it allows for per-minute checks instead of every 5 minutes, and can check for actual page content, not just server response. In addition, when it finds an outage, it runs a root cause analysis. So it would actually tell you that a 500 error caused the check failure (as opposed to server timing out, which is a different problem). No other affiliation.
-
Paul - Thanks for a new way to check and understand all this.
So, if I was able to visit the page just fine normally, and after setting the user agent to Googlebot, then I should be good? I never saw a 500 server error while visiting the page, just in Webmaster Tools. It was dated 2 days ago, but there have been other server error warnings over the past month or two in GWT, so maybe it is a resolved issue.
Can you suggest a method to confirm the overall proper functioning of the .htaccess code? Is there a tool you use to validate your .htaccess code? I checked response headers in Firebug and found all 200 OKs and 304s for images (from the expires header I assume) so to my amateur viewpoint, it looks good. I just don't want to tank the site unwittingly. Obviously not.
-
To note, Kimberly - Webmaster tools keeps a historical record of issues. It may be showing you server error that occurred in the past, but is no longer a problem. Easiest way is to test the URL it is reporting as having problems.
First test by visiting the URL using a regular browser. Then revisit using a regular browser, but with the user-agent set to imitate the Googlebot crawler since it's Googlebot that's reporting the error. (You can do this using the Set User Agent tool built into the Moz Firefox toolbar, or others. It's a critical capability to have for many purposes.) It's possible for the Googlebot to have issues even if a regular visitor sees none, so you want to test for both.
Assuming these tests return the 500 server error, just briefly rename the pertinent htaccess file for a minute, then go back and rerun the tests. If the error goes away with the htaccess disabled, you know the source of the problem lies in the htaccess rules. If the problem persists, you can be pretty certain it's not the htaccess causing it.
Make sense?
Paul
-
Kimberly,
It can, but without which 5XX it is, it is harder to diagnose. (Is it an endless loop, or something else)
I would suggest (based on you trying to redirect what appears to be homepage whether or not the request is for asp or html) this help from Apache. It is a bit deep, but you appear to want to do it yourself and this is a resource I would suggest.
If you look about a third down the page there is a content box that covers tons of variables.
Best,
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Would You Redirect a Page if the Parent Page was Redirected?
Hi everyone! Let's use this as an example URL: https://www.example.com/marvel/avengers/hulk/ We have done a 301 redirect for the "Avengers" page to another page on the site. Sibling pages of the "Hulk" page live off "marvel" now (ex: /marvel/thor/ and /marvel/iron-man/). Is there any benefit in doing a 301 for the "Hulk" page to live at /marvel/hulk/ like it's sibling pages? Is there any harm long-term in leaving the "Hulk" page under a permanently redirected page? Thank you! Matt
Intermediate & Advanced SEO | | amag0 -
Can some sort of wildcard redirect be used on a single folder path?
We have a directory with thousands of pages and we are migrating the entire site to another root URL. These folder paths will not change on the new site, but we don't want to use a wildcard to redirect EVERYTHING to the same folder path on the new site. Setting up manual 301 redirects on this particular directory would be crazy. Is there a way to isolate something like a wildcard redirect to apply only to a specific folder? Thanks!
Intermediate & Advanced SEO | | MJTrevens0 -
Does removal of internal redirects(301) help in SEO
I am planning to completely remove 301 redirects manually by replacing such links with actual live pages/links. So there will be no redirects internally in the website. Will this boost our SEO efforts? Auto redirects will be there for incoming links to non-existing pages. Thanks, Satish
Intermediate & Advanced SEO | | vtmoz0 -
Switching from HTTP to HTTPS: 301 redirect or keep both & rel canonical?
Hey Mozzers, I'll be moving several sites from HTTP to HTTPS in the coming weeks (same brand, multiple ccTLDs). We'll start on a low traffic site and test it for 2-4 weeks to see the impact before rolling out across all 8 sites. Ideally, I'd like to simply 301 redirect the HTTP version page to the HTTPS version of the page (to get that potential SEO rankings boost). However, I'm concerned about the potential drop in rankings, links and traffic. I'm thinking of alternative ways and so instead of the 301 redirect approach, I would keep both sites live and accessible, and then add rel canonical on the HTTPS pages to point towards HTTP so that Google keeps the current pages/ links/ indexed as they are today (in this case, HTTPS is more UX than for SEO). Has anyone tried the rel canonical approach, and if so, what were the results? Do you recommend it? Also, for those who have implemented HTTPS, how long did it take for Google to index those pages over the older HTTP pages?
Intermediate & Advanced SEO | | Steven_Macdonald0 -
Images Returning 404 Error Codes. 301 Redirects?
We're working with a site that has gone through a lot of changes over the years - ownership, complete site redesigns, different platforms, etc. - and we are finding that there are both a lot of pages and individual images that are returning 404 error codes in the Moz crawls. We're doing 301 redirects for the pages, but what would the best course of action be for the images? The images obviously don't exist on the site anymore and are therefore returning the 404 error codes. Should we do a 301 redirect to another similar image that is on the site now or redirect the images to an actual page? Or is there another solution that I'm not considering (besides doing nothing)? We'll go through the site to make sure that there aren't any pages within the site that are still linking to those images, which is probably where the 404 errors are coming from. Based on feedback below it sounds like once we do that, leaving them alone is a good option.
Intermediate & Advanced SEO | | garrettkite0 -
How do you 301 redirect URLs with a hashbang (#!) format? We just lost a ton of pagerank because we thought javascript redirect was the only way! But other sites have been able to do this – examples and details inside
Hi Moz, Here's more info on our problem, and thanks for reading! We’re trying to Create 301 redirects for 44 pages on site.com. We’re having trouble 301 redirecting these pages, possibly because they are AJAX and have hashbangs in the URLs. These are locations pages. The old locations URLs are in the following format: www.site.com/locations/#!new-york and the new URLs that we want to redirect to are in this format: www.site.com/locations/new-york We have not been able to create these redirects using Yoast WordPress SEO plugin v.1.5.3.2. The CMS is WordPress version 3.9.1 The reason we want to 301 redirect these pages is because we have created new pages to replace them, and we want to pass pagerank from the old pages to the new. A 301 redirect is the ideal way to pass pagerank. Examples of pages that are able to 301 redirect hashbang URLs include http://www.sherrilltree.com/Saddles#!Saddles and https://twitter.com/#!RobOusbey.
Intermediate & Advanced SEO | | DA20130 -
Reverse Proxy better than 301 redirect?
Are reverse proxies that much better than 301 redirects? Should I invest the time in doing this? I found out about reverse proxies here: http://www.seomoz.org/blog/what-is-a-reverse-proxy-and-how-can-it-help-my-seo
Intermediate & Advanced SEO | | brianmcc0 -
Any way to find which domains are 301 redirected to competitors' websites?
By looking at the work from an SEO collegue it became clear that his weak linkbuilding graph probably is not the cause for his good rankings for a pretty competitive keyword. (also no social mentions where found) I was wondering what it could be, site structure and other on page optimization factors seems to be ok and I don't think there will be exceptionally good or bad user behavior... Finally I looked at the competitors and found that they have more links, better content en better design, so I got a little stuck. The only reason I can think of is that he is doing 301 redirects (or is rel=canonical tags). Is there a way to trace these redirects back to the source in order to include this important variable in your competitor research? thnx
Intermediate & Advanced SEO | | djingel10