Caps in URL creating duplicate content

[JohnBerger]

Im getting a bunch of duplicate content errors where the crawl is saying

www.url.com/abc has duplicate at www.url.com/ABC

The content is in magento and the url settings are lowercase, and I cant figure out why it thinks there is duplicate consent.  These are pages with a decent number of inbound links.

[JohnBerger]

I checked and it is a magento feature to rewrite caps to lower case.

I added this to htaccess anyway

<code>RewriteMap lc int:tolower
RewriteCond %{REQUEST_URI} [A-Z]
RewriteRule (.*) ${lc:$1} [R=301,L]</code>

One last question before I take this question to a magento forum - how can I look at a page with a caps URL and lower URL and see if they are really different pages or link to the same address.

When you change random letters to caps in our site it sends you to the right page but my browser still shows the mixed caps url instead of replacing with an all lower url - but is that really a different page or is the browser just not changing the caps display when it is really getting the lower case page
```

[igor.pinchevskiy]

Hi John,

I checked the URL you sent me. You do have duplicate pages:

http://www.madebysurvivors.com/destiny

http://www.madebysurvivors.com/DESTINY

both work and return the same page..

I also tried clicking on other links on your site, and then just changing a few letters to the upper case something like this

http://www.madebysurvivors.com/LEArn-human-trafficking-slavery

and it returns the same page

From what I can tell its one of the features in Magento that is making this possible. I would go into settings and disable that setting that forces Magento to use lower case.

Then test it make sure that you DO get a 404 page if you change the letter case on any of your links. Once you test it and you do get a 404 page.

I'm not familiar with Magento so not sure if it has that option or not, but many CMS and ecommerce platforms have a field where you can specify the URL for that page, I would change that field to all lower case.

Test it again, if it works there is one more step that you have to do if you want to keep the same juice from the pages that had the uppercase URL.

You need to duplicate your pages, but you need to make sure that the URL address is the same as it was before (in all CAPS) and then do a 301 redirect to the new page which is in lower case.

Hope this helps and makes sense.

[deltasystems]

This is intended functionality in Magento. It's supposed to help the user experience, as a user can navigate to a page even if they aren't sure on the casing of the words.

Of course that's bad for SEO. You'll need to put in the concept of canonicalization. Here's a free extension by Yoast:

http://www.magentocommerce.com/magento-connect/canonical-url-for-magento.html

Cheers.

Update: seeing your response, your solution of putting in redirects wouldn't be possible. You'd have to cover all combinations of caps/non-caps, and well, that's more work than you should want :). As for why this happens, the uppercase character is being lowercased when checking if something in the database matches the URL. Again, this is intended functionality.

[JohnBerger]

Looks like I do need some more help.

I get a redirect loop if I enter a redirect from

http://www.madebysurvivors.com/DESTINY

to

http://www.madebysurvivors.com/destiny

but I checked and there is no redirect the other way in our database or htaccess.

If I leave the redirect off I get duplicate content - but in the CMS parts of magento there is only one table for this page.

[JohnBerger]

I actually moved all the content from a drupal install so I dont have that many URLs that have the problem.  It looks like the faster way to do this is just redirects the caps to lower case as thats what we use elsewhere..

I dug into the underlying database and cant find any duplicate entries for these pages or odd redirects so I have no idea of the cause.

For some of the pages I think you are right that magento is moving caps down to lower, but there are a few others where it is lower to caps - but it was caps in the drupal site.

Anyway -good to know google sees them differently so Ill put in redirects.  Its only about 20 pages

[igor.pinchevskiy]

Hello John,

If you can provide us with a URL we might be able to dig in to see what is going on. Without it its almost impossible to tell. Also it doesn't matter if you have a decent number of inbound links, duplicate content only refers to pages with similar content. I'm not familiar with Magento platform so this is just a guess, when you created (or imported) pages or categories in Magento originally were they lowercased? If not its possible that Magento added them as all in CAPS and Magento might be forcing it to lower case, therefore you might have duplicates, but once again this is just a guess and without a URL to your site I doubt that someone will be able to help you further.

[AndreVanKets]

www.url.com/abc and www.url.com/ABC  are two completely different pages according to Google

I would redirect any and all pages with capitals to the corresponding lower case URL's.

Dont worry about the link juice as it will pass over via the redirect. It will also be much better than having 2 identical pages competing with eachother (according to Google)

Greg