Can an incorrect 301 redirect or .htaccess code cause 500 errors?
-
Google Webmaster Tools is showing the following message:
_Googlebot couldn't access the contents of this URL because the server had an internal error when trying to process the request. These errors tend to be with the server itself, not with the request. _
Before I contact the person who manages the server and hosting (essentially asking if the error is on his end) is there a chance I could have created an issue with an incorrect 301 redirect or other code added to .htaccess incorrectly?
Here is the 301 redirect code I am using in .htaccess:
RewriteEngine On
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/.]+/)*(index.html|default.asp)\ HTTP/
RewriteRule ^(([^/.]+/)*)(index|default) http://www.example.com/$1 [R=301,L]
RewriteCond %{HTTP_HOST} !^(www.example.com)?$ [NC]
RewriteRule (.*) http://www.example.com/$1 [R=301,L]
Could adding the following code after that in the .htaccess potentially cause any issues?
BEGIN EXPIRES
<ifmodule mod_expires.c="">ExpiresActive On
ExpiresDefault "access plus 10 days"
ExpiresByType text/css "access plus 1 week"
ExpiresByType text/plain "access plus 1 month"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
ExpiresByType application/x-javascript "access plus 1 month"
ExpiresByType application/javascript "access plus 1 week"
ExpiresByType application/x-icon "access plus 1 year"</ifmodule>END EXPIRES
(Edit) I'd like to add that there is a Wordpress blog on the site too at www.example.com/blog with the following code in it's .htaccess:
BEGIN WordPress
<ifmodule mod_rewrite.c="">RewriteEngine On
RewriteBase /blog/
RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /blog/index.php [L]</ifmodule>END WordPress
Thanks
-
Just to follow up on your last question about 404s, Kim...
No, having a bunch of 404s like that will be no more work for the server than if they were landing on actual blog pages - in fact somewhat less work as the 404 page generally has less content and far fewer database calls.
Also, a page timing out due to server load (server working too hard) doesn't generally result in a 500 error, it just returns a timed-out error. 500 errors are delivered when something actually breaks the server's ability to deliver the correct page content.
Paul
-
Wow, you are very quickly and easily making me much better at what I do:) Thanks for that.
I actually just updated the code a couple days ago by adding the Expires code and fixing the redirect. Maybe the previous double 301 redirect could be the culprit? Or - something I mentioned in another question - there were a ton of 404s because of a blog that wasn't redirected to the /blog subdirectory correctly, which I fixed recently. Could something like that cause the server to work to hard and return a 500 server error?
I'll definitely check out the logs and Pingdom.
Great information and advice.
-
Sorry - and to be clear about your htaccess testing question - no there's no "tool" I've ever heard of. You test it by doing exactly as you've done - ensuring that pages respond correctly and with correct headers. Then you implement a monitoring system to ensure that you know every time that correct behaviour fails. That way you can get the site back up quickly, and have a record of when & how often it happened so you can properly troubleshoot if you have an issue.
Three troubleshooting steps
- become aware as soon as there is a problem
- fix the problem asap to minimize impact on users
- investigate and fix the root cause so it doesn't happen again.
All of these steps depend on a monitoring/alerting system, otherwise you'll always be behind the curve and/or working in the dark.
Hope that helps?
Paul
-
Great answer Paul.
-
As far as I understand, Kimberly, you've only changed the htaccess in the last day or 2? in which case the server error would have been from before your updates.
As far as monitoring - you can check the server error logs (via FTP or in cPanel if that's what the hosting account uses) to check for frequent 500-level server errors.
In addition, I strongly recommend that all commercial sites must have uptime monitoring in place. I like to use Pingdom's paid basic plan which allows monitoring of up to 10 pages. I then select a number of relevant pages and set the tool to test each page, and to check for an actual text snippet on each page (using their custom settings). I monitor things like the home page, the blog home page, a blog post, a blog category page, and critical call-to-action pages. Basically different types of page templates that might respond differently to server issues. plus critical money-making pages.
This way, Pingdom will alert you immediately any time those pages don't respond normally (like when a server gives back a 500 error, or the server goes unresponsive due to overload etc). Monitoring these pages every minute is the ONLY way to really know whether your server and website software are performing properly and consistently. This is a critical component of any professionally run website, in my opinion.
Often Pingdom confirms that things are running fine, but I literally can't count the number of times I've instituting uptime monitoring for new clients, only to find the site has huge downtime no one was really aware of, because they just aren't on their own site often enough to know when it's down. (And you certainly shouldn't be relying on customers to inform you the site has issues. By then it's FAR too late.)
Paul
P.S There are certainly other uptime monitoring systems out there, some are even free. I recommend Pingdom because I've used it for years and it's been consistently excellent. Also, it allows for per-minute checks instead of every 5 minutes, and can check for actual page content, not just server response. In addition, when it finds an outage, it runs a root cause analysis. So it would actually tell you that a 500 error caused the check failure (as opposed to server timing out, which is a different problem). No other affiliation.
-
Paul - Thanks for a new way to check and understand all this.
So, if I was able to visit the page just fine normally, and after setting the user agent to Googlebot, then I should be good? I never saw a 500 server error while visiting the page, just in Webmaster Tools. It was dated 2 days ago, but there have been other server error warnings over the past month or two in GWT, so maybe it is a resolved issue.
Can you suggest a method to confirm the overall proper functioning of the .htaccess code? Is there a tool you use to validate your .htaccess code? I checked response headers in Firebug and found all 200 OKs and 304s for images (from the expires header I assume) so to my amateur viewpoint, it looks good. I just don't want to tank the site unwittingly. Obviously not.
-
To note, Kimberly - Webmaster tools keeps a historical record of issues. It may be showing you server error that occurred in the past, but is no longer a problem. Easiest way is to test the URL it is reporting as having problems.
First test by visiting the URL using a regular browser. Then revisit using a regular browser, but with the user-agent set to imitate the Googlebot crawler since it's Googlebot that's reporting the error. (You can do this using the Set User Agent tool built into the Moz Firefox toolbar, or others. It's a critical capability to have for many purposes.) It's possible for the Googlebot to have issues even if a regular visitor sees none, so you want to test for both.
Assuming these tests return the 500 server error, just briefly rename the pertinent htaccess file for a minute, then go back and rerun the tests. If the error goes away with the htaccess disabled, you know the source of the problem lies in the htaccess rules. If the problem persists, you can be pretty certain it's not the htaccess causing it.
Make sense?
Paul
-
Kimberly,
It can, but without which 5XX it is, it is harder to diagnose. (Is it an endless loop, or something else)
I would suggest (based on you trying to redirect what appears to be homepage whether or not the request is for asp or html) this help from Apache. It is a bit deep, but you appear to want to do it yourself and this is a resource I would suggest.
If you look about a third down the page there is a content box that covers tons of variables.
Best,
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 redirects and impact on page authority
I need to restructure a section of my website, changing some page titles and moving some pages to other sections. This will then change the URLs but the CMS I use will automatically create 301 redirects so the old URLs still work. The question is, will this have any negative impacts on page authority/page rank? From what I've read, it seems having 301's used to have a negative impact but doesn't anymore?
Intermediate & Advanced SEO | | ciehmoz0 -
Using a Reverse Proxy and 301 redirect to appear Sub Domain as Sub Directory - what are the SEO Risks?
We’re in process to move WordPress blog URLs from subdomains to sub-directory. We aren’t moving blog physically, but using reverse proxy and 301 redirection to do this. Blog subdomain URL is https://blog.example.com/ and destination sub-directory URL is https://www.example.com/blog/ Our main website is e-commerce marketplace which is YMYL site. This is on Windows server. Due to technical reasons, we can’t physically move our WordPress blog to the main website. Following is our Technical Setup Setup a reverse proxy at https://www.example.com/blog/ pointing to https://blog.example.com/ Use a 301 redirection from https://blog.example.com/ to https://www.example.com/blog/ with an exception if a traffic is coming from main WWW domain then it won’t redirect. Thus, we can eliminate infinite loop. Change all absolute URLs to relative URLs on blog Change the sitemap URL from https://blog.example.com/sitemap.xml to https://www.example.com/blog/sitemap.xml and update all URLs mentioned within the sitemap. SEO Risk Evaluation We have individual GA Tracking ID and individual Google Search Console Properties for main website and blog. We will not merge them. Keep them separate as they are. Keeping this in mind, I am evaluating SEO Risks factors Right now when we receive traffic from main website to blog (or vice versa) then it is considered as referral traffic and new cookies are set for Google Analytics. What’s going to happen when its on the same domain? Which type of settings change should I do in Blog’s Google Search Console? (A). Do I need to request “Change of Address” in the Blog’s search console property? (B). Should I re-submit the sitemap? Do I need to re-submit the blog sitemap from the https://www.example.com/ Google Search Console Property? Main website is e-commerce marketplace which is YMYL website, and blog is all about content. So does that impact SEO? Will this dilute SEO link juice or impact on the main website ranking because following are the key SEO Metrices. (A). Main website’s Avg Session Duration is about 10 minutes and bounce rate is around 30% (B). Blog’s Avg Session Duration is 33 seconds and bounce rate is over 92%
Intermediate & Advanced SEO | | joshibhargav_200 -
Can’t put a finger on, what is causing 12 year domain, SEO optimized and decent link profile to rank lower than other less superior domains.
Can’t put a finger on, what is causing 12 year domain, SEO optimized and decent link profile to rank lower than other less superior domains. I have dissected the site and link, content, etc profile using ahrefs tools, still no luck, and unfortunately they do not have a community to ask anyone opinion. Hoping someone on Moz will be able to provide me with a secondary opinion or something I obviously missing here. Looking for any constructive feedback/professional opinion with fresh look on what maybe the cause of our down rankings and what may be a cause of it. Any feedback is very much appreciated. Search Term: 3030 aventura condos / One of our link samples (SE Position #6): https://goo.gl/FbYj4V Competing Domains (SE Position #1): https://goo.gl/fLPKX5 Competing Domains (SE Position #2): https://goo.gl/GqXGse
Intermediate & Advanced SEO | | Im_Jake0 -
301 Redirecting from Static to Dynamic URLs. I think we messed up
I'm looking for some guidance on an issue I believe we created for ourselves and if we undo what we did. We recently added attributed search to our sites. This of course created a bunch of dynamically generated URLS. For various reasons, it was decided to take some of our existing static URLs and 301 redirect them to their dyanamic counterpart. Ex .../Empire-Paintball-Masks-0Y.aspx now redirects to .../Paintball-Masks-And-Goggles-0Y.aspx?Manufacturer=Empire Many of these stat URLS had top 3 rankings for their associated keywords. Now, we don't rank for anything. I realize that 301 redirecting is the way to go...if you NEED to. My guess is our drop in keyword ranking is directly tied to what we did. I'm looking for an solid argument to be made to my boss as to why we should not have done this and that it, more than likely has resulted in dropped keyword rankings and organic traffic. I welcome any input. Also, if we decided to revert back (remove all 301 redirects and de-index all dynamic URLS), what is the likely hood we can recapture some of this lost organic traffic? Can I disallow indexing in a robot.txt file to remove, say anything with a '?' in the URL? Would the above URL example (which was ranking in the top 3 in SERPs), have a good chance of finding its way back? thanks
Intermediate & Advanced SEO | | Istoresinc1 -
Can I redirect duplicate blogs to give credit to one?
I have two sites that have no duplicate content (yet). One ranks better than the other but has a crappy hyphenated domain name (Domain A), and the other one is the "brand site" with a better domain name (Domain B). I'm creating a blog with technical articles and corresponding videos. I want the videos to refer to the better domain name (Domain B) because I can't see referring people to a hyphenated domain (it would sound horrible). But, the hyphenated domain has a better chance of improving it's rankings (long story why). Can I duplicate the content and just use a canonical tag on Domain B to give the credit to Domain A? If I do that, is it done on each post? Or the blog's main page? What I think would happen is any links to Domain B would pass the juice to Domain A. Is that correct? I know Canonical's are tricky and I don't want to screw this up, so I'd greatly appreciate some advice from the experienced people on here. Thank you.
Intermediate & Advanced SEO | | PhoenixDev0 -
Restructuring/Removing 301 Redirects Due To Newly Optimized Keywords
Just to be clear, this is for one unique page on a website. Also, please see my diagram attached. Let's say that a page's URL was originally /original. So, you optimize the page for a new keyword (keyword 1), and therefore change the URL to /keyword-1. A 301 redirect would then be placed... /original > /keyword-1 However, let's say 6 months down the road you realize that the keyword you optimized the page for (keyword 1) just isn't working. You research for a new keyword, and come up with (keyword 2). So, you'd like to rename the page's URL to /keyword-2. After placing a redirect from the current page (keyword 1) to the 'now' new page (keyword 2), it would look like this... /original > /keyword-1 > /keyword-2 We know that making a server go through more than one redirect slows the server load time, and even more 'link-juice' is lost in translation. Because of this, would it make sense to remove the original redirect and instead place redirects like this? /original > /keyword-2 /keyword-1 > /keyword-2 To me, this would make the most sense for preserving SEO. However, I've read that removing 301 redirects can cause user issues due to browsers caching the now 'removed' redirect. Even if this is ideal for SEO, could it be more work than it's worth? Does anyone have any experience/input on this? If so, I greatly appreciate your time! oDvLl.jpg
Intermediate & Advanced SEO | | LogicalMediaGroup1 -
Confusing 301 / Canonical Redirect Issue - Wizard Needed
I had two pages on my site with identical content. What I did was 301 redirect one page to the other. I also added canonical redirect code to the page that held the 301 code. Here is what I have: www.careersinmusic.com/music-colleges.aspx - this page was a duplicate and I needed it to resolve to:
Intermediate & Advanced SEO | | 4Buck
www.careersinmusic.com/music-schools.aspx Here is the code I used: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX music-colleges.aspx
<%@ Page Language="VB" AutoEventWireup="false" CodeFile="music-colleges.aspx.vb" Inherits="music_colleges" %>
http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
http://www.w3.org/1999/xhtml"> http://www.careersinmusic.com/music-schools.aspx"/> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
music-colleges.aspx.vb
Partial Class music_colleges
Inherits System.Web.UI.Page
Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
Response.Status = "301 Moved Permanently"
Response.AddHeader("Location", "http://www.careersinmusic.com/music-schools.aspx")
End Sub
End Class XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX The problem:
For some reason, when the search “music colleges” is done in Google, I am #7. When the term “music schools” is done, I am around 119. I MUST be getting a penalty for some reason, I just cannot figure the reason. When perform well for one term and terrible for the next? All I can come up with is a duplicate content penalty or something along those lines. Also, music-colleges.aspx seems to still be in Googles index, even though the above 301 happened months ago. Thoughts? site:www.careersinmusic.com/music-colleges.aspx Any insight into this would be GREATLY appreciated. Many Thanks!0 -
Why isnt my crawl results showing a 301 redirect even though I have a 301 rewrite in my .htaccess file?
Ive searched the previous Q&A's & cant find an answer so I;ll ask it here 🙂 crawling my site shows isnt the 301 redirect that i have from my non www to my www domainIts only showing all the results for my www subdomain.As i'm new to SEO & SeoMoz I dont fully understand. Any help would be greatly appreciated because my site is like 2 & a half years old & i'm trying to learn seo so I can rank higher in the serp's. Thanks
Intermediate & Advanced SEO | | PCTechGuy20120