301 Redirect with index.asp

[TroyW]

I am very new to all of this so forgive the newbie questions I will get better.   Ok so after starting a campaign I see that I have many issues including where some pages are being deemed as duplicate content.

1. The report says the http://lucid8.com has duplicate content on 2 other pages

2. When I look at them it shows that http://lucid8.com/index.asp and http://www.lucid8.com are duplicates.

3. Really these are the exactly the same page because the default page that is opened for www.lucid8.com http://www.lucid8.com etc always opens the index.asp page.

4. Now I read that I should do permanent redirects and how to do this via IIS and I tried to do a redirect from index.asp to www.lucid8.com but that does not work because www.lucid8.com is pointing to index.asp and so we end up in a circle.

So the question is how do I get rid of these duplicate page references without causing problems.

Thanks

[TroyW]

Yeah a major pain and went round and round since it was happening on all machines.  Once I got a machine that didn't do it it got me thinking..

Anyway hope it helps others in the future

[MichaelC-15022]

That's brutal!  I've seen something like that happen watching the HTTP responses via httpfox sometimes, where I've made a change on the server to a redirection and not seeing it in the browser.

[TroyW]

Hey thought that I would also mention that the issue where it was referring from Lucid8.com to Https://Lucid8.com was actually a bug in firefox.  Apparently FF has a bug where IF you ever went to say Https://Lucid8.com in the past, even though you enter lucid8.com it will autocorrect upon submission and send you to Https://Lucid8.com

They say that removing the HTTPS items from history will fix it, but in my case it did not and instead I had to export all my bookmarks, kill off the profile and then import the bookmarks again and now all is well.

[MichaelC-15022]

Good, I'm glad you got this all working.  MC

[TroyW]

ok so I figured out a workaround since it appears that because I am on IIS6 the web.config is a but finicky aka may or may not work for redirection.  Anyway I used the information in this link http://johnnycode.com/2012/01/04/in-iis6-http-301-redirect-from-non-www-to-www/ to create a now non-www site and point it to the www site and all seems to be working well, expect if a person puts in a HTTPS within the WWW but that should be a really rare event so I am not going to hassle with it.    I also got rid of the relative URLS on the website and made them all explicit.

Fingers crossed that next run will show this issue is cleaned up....

Thanks for all your help and patience

[MichaelC-15022]

Sounds like something in a web.config file.  Is there possibly a web.config file in a higher-level folder?

Or, perhaps there's an IIS plugin that you're using that is doing this?

[TroyW]

Yup something very strange.   So it appears that

1. the redirect code in the web.config does not work

2. I may have something else going on because whenever I enter lucid8.com it redirects to https://www.lucid8.com so wondering if you have any ideas as to how I might track down WHAT exactly is making the page change to from Lucid8.com to https://www.lucid8.com???

[MichaelC-15022]

Yes, that looks to me like 2 clues that both say that the web.config isn't being read for some reason.

[TroyW]

Hmm well no all seems to be working well here, although the code in the web.config doesn't seem to be working for redirects, i.e. I can put in lucid8.com and it does not change to www.lucid8.com so maybe its not picking up the changes in web.config?

[MichaelC-15022]

If your default document is index.asp, and you're redirecting index.asp to /, then I would expect you'd get an infinite redirection loop.

[TroyW]

Sorry for the delayed response, been buried putting out other fires.  That said

1. The web.config  has the following in place so it refers to /

<rewritemaps><rewritemap name="Redirect"><add key="/index.asp" value="/"></add></rewritemap></rewritemaps>

2. All of my menu systems and links now reference the home page as www.lucid8.com or as / which I assume should not be an issue?

3. The only issue outstanding is your recommendation to

A: copy index.asp to home.asp

B: then would I strip the entire contents from index.asp and put some type of code within to tell it to look at home.asp?  if so any help you have here would be appreciated.

NOTE:  I am still a bit lost as to why I would have to do 3.a & b since steps 1 & 2 would seem to resolve the issue, but perhaps I am missing a critical point?

[TroyW]

Sorry for the delay in response, had another fire to deal with....

Anyway I put this in place and it just doesn't seem to work, i.e.

1. if I put in lucid8.com I would expect it to correct itself to www.lucid8.com or http://www.lucid8.com however instead it just opens the page and keeps lucid8.com in the URL.   Any ideas here?

2. I also notice that from that point on when I click to navigate to another place on the site that the URL stays without the www and I am going to guess that is because some of the pages have ABSOLUTE paths, i.e. www.lucid8.com/purchase/login.asp while others are relative, i.e /purchase/login.asp

So I guess two questions here i.e.

A: any idea how to track down why the redirect in the web.config is not working?

B: if we get # 1 working will it solve the # 2 issue or do we need to change all relative paths to absolute?

thanks

[TroyW]

Excellent Info

thanks

[MichaelC-15022]

You're pretty much on track with this now.  The key points are:

• nothing outside of your web config file should be aware of what your home page's file name actually is
• Googlebot, your menu, etc. should all see your home page as /
• any existing links to index.asp should be 301 redirected to / to consolidate the link juice

And, the reason you're renaming index.asp to something else is that otherwise you'd have an infinite redirection loop of index.asp -> / -> gets resolved to index.asp -> / -> ....

[MichaelC-15022]

I'll second Mike's comment about using Screaming Frog to test immediately.

The only thing I'll add is that it's worth doing an httpfox test of one example page to make sure you don't have a chain of redirects from your combinations of rules--Screaming Frog will just show you 1 of the response codes per page (it's unclear to me if it's the first or the the last response code).  You'll catch things like one of the redirects in the chain being a 302 not a 301 (for instance, a basic Response.Redirect in the ASP code itself does a 302, not a 301!).

[Mike.Goracke]

MatchAll means that all of the conditions must be met for the rule to process. It has nothing to do with case sensitivity.

Give this a whirl (I added your example of including a purchase folder in the rule):

<system.webserver><rewrite><rewritemaps><rewritemap name="Redirect"><add key="/index.asp" value="/"></add></rewritemap></rewritemaps>
        <rules><rule name="CanonicalHostNameRule" enabled="true"><match url="(.)"><conditions logicalgrouping="MatchAll" trackallcaptures="false"><add input="{HTTPS}" pattern="^ON$"> 
                    <add input="{HTTP_HOST}" pattern="^www.lucid8.com$" negate="true"></add></add></conditions>
                <action type="Redirect" url="<a href=" http:="" www.lucid8.com="" {r:1"="">http://www.lucid8.com/{R:1}" redirectType="Permanent"  />
           </action></match></rule>
            <rule name="AllHTTPexceptSIGNIN" stopprocessing="true"><conditions><add input="{HTTP_HOST}" pattern="<a href=" http:="" (www.)(.)$"="">http://(www.)(.*)$" negate="true" />
                    <add input="{HTTPS}" pattern="on"><add input="{URL}" pattern="download" negate="true"><add input="{URL}" pattern="purchase" negate="true"></add></add></add></add></conditions>
                <action type="Redirect" url="<a href=" http:="" {http_host}="" {r:0"="">http://{HTTP_HOST}/{R:0}" redirectType="Permanent"/>
           </action></rule></rules></rewrite></system.webserver>

[TroyW]

ok thats  great information since we only have a few sections where HTTPS is enforced and the cleaner I can make things the better.

That said here is what I have now in my web.config file so

1. how would I best integrate what you have above with this?

2. As I add new sections for HTTPS is it a single line for each new pattern, i.e. like "purchase" for example.

3. Am I correct that the "MatchAll" is for non case sensitivity?

<system.webserver><rewrite><rewritemaps><rewritemap name="Redirect"><add key="/index.asp" value="/"></add></rewritemap></rewritemaps>
            <rules><rule name="CanonicalHostNameRule" enabled="true"><match url="(.*)"><conditions logicalgrouping="MatchAll" trackallcaptures="false"><add input="{HTTPS}" pattern="^ON$"><add input="{HTTP_HOST}" pattern="^www.lucid8.com$" negate="true"></add></add></conditions>
                    <action type="Redirect" url="http://www.lucid8.com/{R:1}" redirecttype="Permanent"  =""></action></match></rule></rules></rewrite></system.webserver>

[TroyW]

Excellent information and it appears that today is my next crawl date.  I will also take a look at screaming frog to see what it is all about.  Thanks much for all your help and insight its invaluable.  I am stuck trying to do this myself because of budgetary reasons and also my past experience with so called SEO experts was not so great...  Lots of posers out there, however tools like this and these message boards make it much better and also easier to find other people to work with in the future.

Thanks again Mike

Troy

[Mike.Goracke]

If you go to your campaign overview page, you will see a little box below the mini overviews that will say something like, "Last Crawl Completed: Apr. 3rd, 2013 Next Crawl Starts: Apr. 10th, 2013"

I personally use SEOmoz PRO tools in combination with Screaming Frog. I verify many of the problems using Screaming Frog, then fix them, then rescan (which is instant), then wait for my SEOmoz PRO tools to reflect my changes. SEOmoz does a great job of warning you and keeping you in the know... where Screaming Frog gives you a lot of information, but you really have to know what you are looking for and would have to keep on top of it... SEOmoz is more automated... if that makes sense.

Mike

[Mike.Goracke]

The only thing that can happen is if Google indexes an http and https version of the same page. It isn't a HUGE deal... just depends on how obsessed you are about the site structure.

You could potentially have visitors start linking to the https version vs the http version, in which case that would be a problem.

Depending on the complexity of your site and code, this rule may be able to help you... where it says, pattern="download" that means that if someone visits the downloads section of your website, it will allow for https.  You can continue to add additional folders by just repeating that code and replacing "download" with whatever the folder names are that you want to allow to use HTTPS. I did a Google search of site:lucid8.com inurl:https and noticed that your download section was indexed as using HTTPS, that is why I used it in this example.

If you are just using simple folder structures, this rule is not too bad to implement. I previously just implemented it with pattern matching and that was not fun.

Any way, if it helps great, if not, just try to keep your internal linking as consistent as possible. Sometimes the best way to do this is use absolute paths vs relative.

Mike

<rule name="AllHTTPexceptSIGNIN" stopprocessing="true"><conditions><add input="{HTTP_HOST}" pattern="<a href=" http:="" (www.)(.)$"="">http://(www.)(.)$" negate="true" />
                        <add input="{HTTPS}" pattern="on"><add input="{URL}" pattern="download" negate="true"></add></add></add></conditions>
                    <action type="Redirect" url="<a href=" http:="" {http_host}="" {r:0"="">http://{HTTP_HOST}/{R:0}" redirectType="Permanent"/>
               </action></rule>

[TroyW]

So the weekly scan from SEOmoz, will that happen one week from the original scan date or?  And if so will the campaign show when the scan has been updated or will I get a notification that its been updated or??

Just trying to figure out how to best use the tool and how long to wait for changes to appear in SEOmoz before I can mark an issue internally as being resolved so I can move on to the others

[TroyW]

ok so I corrected all the internal references by taking out the explicit use of index.asp so moving forward that would seem to be solved.

However that said is the premise here that index.asp is already seen as a separate page from www.lucid8.com and therefore  should become the reference page to a copy of itself called home.asp or whatever so that anything out there pointing to index.asp will now get consolidated into the new home.asp file and that when that happens it will resolve any issue with the search engines thinking that www.lucid8.com and www.lucid8.com/index.asp are different?

[TroyW]

We do use HTTPS in different sections on the site.  I played with a few things however it appears that unless you get complex there is no easy way to ensure the proper pages go to HTTP or HTTPS so will leave as the default of HTTPS now since it clearly does no harm in opening a non HTTPS page as HTTPS unless there is something I am missing??

[MichaelC-15022]

You can thoroughly test these your self using the Firefox plugin httpfox.

The advantage this particular tool has over pretty much everything else is that you see not only the LAST http response code, but the entire chain.

I've answered private Q&A questions here from people where Google was seeing one of the response codes in the chain, but other tools were seeing a different one.  If I remember correctly, it was a chain of 301s and 302s.

You want (ideally) a SINGLE 301; if you cannot make that happen, then a chain of 301s, with no 302s or 404s in the middle.

[Mike.Goracke]

Cool. Glad to help.

Taking care of https is another rule that you would need to implement. This can be a bit more tricky if your site uses https for other purposes (such as signing in or a checkout process, etc.)

If you do not use https anywhere on your site, I can probably come up with some code that would work for you.

The SEOmoz scan is done on a weekly basis, so there isn't much you can do there. And as for other tools to do these scans, you really cannot get the level of detail in a user-friendly manner that SEOmoz provides - in my opinion.

Mike

[TroyW]

Mike I think that did the trick, which is awesome!!!   I validated the results by entering lucid8.com and it redirected me, however it added HTTPS vs the default of HTTP i.e. i got https://www.lucid8.com/ vs. http://www.lucid8.com/

Therefore;

1. Does the code above need to be modified or do I need to change some other setting

2. Is there a more robust way of validating these resolutions, i.e. if I do a new scan with SEOmoz will it immediately report that the issues have been fixed or ???  OR are there other methods to ensure I fixed these two issues completely?

[Mike.Goracke]

If you are using IIS, then you would want to configure these fixes in your web.config file.

I think you'd want something like this:

<configuration><system.webserver><rewrite><rewritemaps><rewritemap name="Redirect"><add key="/index.asp" value="/"></add></rewritemap></rewritemaps>
            <rules><rule name="CanonicalHostNameRule" enabled="true"><match url="(.*)"><conditions logicalgrouping="MatchAll" trackallcaptures="false"><add input="{HTTP_HOST}" pattern="^www.lucid8.com$" negate="true"></add></conditions>
                    <action type="Redirect" url="http://www.lucid8.com/{R:1}"></action></match></rule></rules></rewrite></system.webserver></configuration>

If that does not work, I would recommend breaking it into sections - for example, try doing the redirect code first, then the canonical code second, or vise versa. I always look up examples of web.config configurations on stackoverflow.com.

Hope this helps Troy.

Mike

[MichaelC-15022]

FYI you'll want to rename index.asp to something else, and then change the default document to point to the new filename (e.g. "home.asp"), then 301 redirect index.asp to /.  This will consolidate any link juice currently going to index.asp onto your one real home page, "/".

When you think you're all done, then use the Firefox plugin HTTPfox to see your entire HTTP response code stream when you hit index.asp, the non-www version of your site, etc.  You're looking to see a single 301 redirect for each.

[MichaelC-15022]

Mike is correct.  You've got 2 separate problems here.  I actually wrote articles on these for About Us here and here.

[TroyW]

BTW I Googled the issue and then poked around IIS on could not find any type of duplicate entry for www vs non www, however perhaps I am missing something simple?   The entry that I have for the main site doesn't seem to say whether its for WWW or Non-WWW either so?

Also read an article that said this might be the cause of a duplicate DNS entry, i.e. one for WWW and one for non WWW??

[TroyW]

Mike sorry for the delay in response....

I am using IIS on a Windows 2003 Server

[Mike.Goracke]

That is a change you will make on your server.

Depending on the backend software you use, the code or setting to implement this type of change will be different.

What software do you use?

Mike

[TroyW]

Great info thanks guys and

1. I found and corrected the /index.asp issue and am searching to ensure that issue is resolved on all other pages as well

2. Regarding the http://lucid8.com and http://www.lucid8.com  I am confused as to where this is set or how to adjust.  Can you provide any additional hints as to how I should address this?

Thanks to both of you this has been a great help!

[jesse-landry]

excellent catch, Mike. He's absolutely correct about the index.asp page.

these two fixes should really help your site.

[Mike.Goracke]

Hi Troy,

The issue with /index.asp is coming up because of your Home link in your footer. Instead of resolving to home "/" it is being pointed to "/index.asp".

And the reason it is also showing the www-version is because when the crawler is on lucid8.com and it crawls the link to the right of customer/partner login "customers", the crawler is brought to www.lucid8.com/downloadplus/Resource_login.asp Then, from here, if you click on any of the hyperlinks, you are brought to the www-version of the page instead of the non-www you specified as your preferred.

Jesse is correct in saying you need to choose either a www or non-www version; however, you also need to fix the /index.asp issue.

Hope this helps.

Mike

[jesse-landry]

i think that because you said "http://lucid8.com/index.asp and http://www.lucid8.com are duplicates" your issue might actually be in the www and non-www canonicalization of your site.

Make sure that you pick one or the other (i recommend www) and in your IIS server settings redirect the non-www to the www. (or vice versa if you must)

This should solve your problem.

**update -- I just went to your site and yes this is happening. You have two sites going, http://lucid8.com and http://www.lucid8.com. The "index.asp" portion of it actually has nothing to do with this. But Google is currently seeing each version as two separate pages, resulting in duplicates of every page on that domain.

Definitely add the 301 redirect in your IIS settings to one or the other. A linkrel=canonical tag wouldn't hurt on each page either, but isn't necessary.

Good luck