Duplicate content
-
I have just ran a report in seomoz on my domain and has noticed that there are duplicate content issues, the issues are:
www.domainname/directory-name/
www.domainname/directory-name/index.php
All my internal links and external links point to the first domain, as i prefer this style as it looks clear & concise, however doing this has created duplicate content as within the site itself i have an index.php page inside this /directory-name/ to show the page.
Could anyone give me some advice on what i should do please?
Kind Regards
-
Hi Gary,
Here's some code from an htaccess file I've used before that solves the issue you've got with index.php at the end of all your urls:
#remove /index.php and ensure admin works okay
RewriteCond %{REQUEST_URI} !^/administrator
RewriteCond %{THE_REQUEST} ^.*/index.php\ HTTP/
RewriteRule ^(.*)index.php$ /$1 [R=301,L]
notice the line that contains ^/administrator , in Joomla, admin login is usuall on http://site.com/administrator/index.php
so, removing the index.php from the admin url would prevent any access to the admin screens! If your cms has a similar url, be sure to replace 'administrator' with the relevant url.
-
Hi Ade,
Thanks for all your help.
I will post a new question on the Q&A Forum regarding the .htaccess rule.
Kind Regards
-
Hi Gary,
That one is a bit beyond me I'm afraid and I am not familiar with WebEdition at all.
With most CMS there are normally either built-in or add-on extensions to help with re-writing your urls but you need to be really careful that you don't end up with a completely new set of urls that don't match either of your originals.
A .htaccess rewrite rule may be your best option but I don't know what the coding for it would be.
-
We are using a CMS, its called WebEdition, is there a technical question i should ask them in what i need to do?
Kind Regards
-
Ahhhh. No definitely not practical, I thought that it was just the one url.
Are you using a content management system for your site such as Joomla?
-
Do you think that's practical to do that?
As i will need to do a 301 on literally every page if i don't want to show the /index.php
Is this what seomoz.org website does? for example:
-
In that case you can just add a 301 redirect in to your .htaccess file below the code you added earlier.
redirect 301 /football-teams/index.php http://www.mydomain.com/football-teams/
-
Hi Ade,
Yes, i tested http://www.mydomain.com/football-teams//index.php however it did not resolve to http://www.mydomain.com/football-teams/
Any ideas?
-
Hi Gary.
Have you tried visiting the url http://www.mydomain.com/football-teams/index.php to see if it now resolves to http://www.mydomain.com/football-teams/ ?
If it does then the issue is fixed, the next time SEOMoz crawls your site the error will dissapear.
Cheers.
Ade.
-
Hi Ade,
Thanks for the speedy reply.
I have now implemented this and works fantastic on the http://www.mydomain.com/
Thank you very much.
There is another issue however, i hope i can make sense here, here goes:
seomoz tool gives me back duplicate content on both these URL's
http://www.mydomain.com/football-teams/
http://www.mydomain.com/football-teams/index.php
I want to use http://www.mydomain.com/football-teams/ as this just look nice & clean.
What would be best practice to fix this issue?
Kind Regards
-
Hey Gary.
Here's the solution that I use.
All my sites are hosted on a linux server so this won't be relevant if your site is hosted on a windows server.
1. create/modify your .htaccess file in your site's root directory.
2. Add the following code to the top of the file:-
RewriteEngine On
RewriteBase /
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /index.php\ HTTP/
RewriteRule ^index.php$ http://www.yourdomain.com/ [R=301,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]RewriteCond %{HTTP_HOST} ^yourdomain.com [NC]
RewriteRule ^(.*)$ http://www.yourdomain.com/$1 [L,R=301]This will ensure that any requests sent to http://yourdomain.com are redirected to http://www.yourdomain.com and that the index.php part of the url is removed.
If you need more help on creating or modifying your .htaccess file then you can find more info here - http://httpd.apache.org/docs/1.3/howto/htaccess.html
All the best.
Ade.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate content or an update ???
Buying Guide and Product Category page competing for the same keyword ? Got a “nuts and bold website” selling basic stuff. Imagine selling simple nuts, bolts and washers (the little ring that goes in between) in different metals. Imagine a website with a very wide and deep line of these simple products. For long tail keywords we rank well (Example: 0.25 inch bolts). For the keyword: “Nuts bolts” our main category page use to rank well low 1<sup>st</sup> page to second page up against the big guys (Amazon, Walmart, Target, Costco, some drug store who may have a mix pack of nuts and bolts, but still Google don’t see the difference and list 2 pages each for these guys). But then in mid-February there were an update and suddenly our “Buying guide for nuts and bolts” rank higher and started to compete with our own product category page. That was never our intention. These two pages now compete for the ranking on page 4<sup>th</sup>. Clearly there were more words on the buying guide page but no changes had been made to it for well months or years. To make up for it some more words were added to the category page, but of cause there is only so many way you can fraise words about “nuts and bolts” without sounding a bit duplicate/re-writing. So what do I do now ?? Clearly the product category page is the one we like to rank highest with the guide a close 2nd. Most customer don’t need the buying guide but it is good to have and great support as we got lot of good comments from customer who read it. Made a link to the buying guide from the category page and wise verses. The category page got an embedded video. Moz list the page authority for the category page to 16 and 1 for the buying guide but clearly G see it differently. Already tried to change the Meta Tag Title and Description a little but it is hard to do if the word “Nuts Bolts” is to appear in the description or people don’t know what to expect. Could just insert a “do not index” for the buying guide but not a good long term solution. Unfortunately I am out of imagination at this point. Any good suggestions ?? Thanks, Kim Any good suggestions ???
Technical SEO | | KimX0 -
Duplicate Page Content and Titles from Weebly Blog
Anyone familiar with Weebly that can offer some suggestions? I ran a crawl diagnostics on my site and have some high priority issues that appear to stem from Weebly Blog posts. There are several of them and it appears that the post is being counted as "page content" on the main blog feed and then again when it is tagged to a category. I hope this makes sense, I am new to SEO and this is really confusing. Thanks!
Technical SEO | | CRMI0 -
Duplicate Content Reports
Hi Dupe content reports for a new client are sjhowing very high numbers (8000+) main of them seem to be for sign in, register, & login type pages, is this a scenario where best course of action to resolve is likely to be via the parameter handling tool in GWT ? Cheers Dan
Technical SEO | | Dan-Lawrence0 -
How different should content be so that it is not considered duplicate?
I am making a 2nd website for the same company. The name of the company, our services, keywords and contact info will show up several times within the text of both websites. The overall text and paragraphs will be different but some info may be repeated on both sites. Should I continue this? What precautions should I take?
Technical SEO | | savva0 -
Duplicate Content for Multiple Instances of the Same Product?
Hi again! We're set to launch a new inventory-based site for a chain of car dealers with various locations across the midwest. Here's our issue: The different branches have overlap in the products that they sell, and each branch is adamant that their inventory comes up uniquely in site search. We don't want the site to get penalized for duplicate content; however, we don't want to implement a link rel=canonical because each product should carry the same weight in search. We've talked about having a basic URL for these product descriptions, and each instance of the inventory would be canonicalized to this main product, but it doesn't really make sense for the site structure to do this. Do you have any tips on how to ensure that these products (same description, new product from manufacturer) won't be penalized as duplicate content?
Technical SEO | | newwhy0 -
Duplicate content problem from an index.php file
Hi One of my sites is flagging a duplicate content problem which is affecting the search rankings. The duplicate problem is caused by http://www.mydomain.com/index.php which has a page rank of 26 How can I sort the duplicate content problem, as the main page should just be http://www.mydomain.com which has a page rank of 42 and is the stronger page with stronger links etc Many Thanks
Technical SEO | | ocelot0 -
Help removing duplicate content from the index?
Last week, after a significant drop in traffic, I noticed a subdomain in the index with duplicate content. The main site and subdomain can be found below. http://mobile17.com http://232315.mobile17.com/ I've 301'd everything on the subdomain to the appropriate location on the main site. Problem is, site: searches show me that if the subdomain content is being deindexed, it's happening really slowly. Traffic is still down about 50% in the last week or so... what's the best way to tackle this issue moving forward?
Technical SEO | | ccorlando0 -
How to publish duplicate content legitimately without Panda problems
Let's imagine that you own a successful website that publishes a lot of syndicated news articles and syndicated columnists. Your visitors love these articles and columns but the search engines see them as duplicate content. You worry about being viewed as a "content farm" because of this duplicate content and getting the Panda penalty. So, you decide to continue publishing the content and use... <meta name="robots" content="noindex, follow"> This allows you do display the content for your visitors but it should stop the search engines from indexing any pages with this code. It should also allow robots to spider the pages and pass link value through them. I have two questions..... If you use "noindex" will that be enough to prevent your site from being considered as a content farm? Is there a better way to continue publication of syndicated content but protect the site from duplicate content problems?
Technical SEO | | EGOL0