Duplicate content penalty
-
when moz crawls my site they say I have 2x the pages that I really have & they say I am being penalized for duplicate content. I know years ago I had my old domain resolve over to my new domain. Its the only thing that makes sense as to the duplicate content but would search engines really penalize me for that? It is technically only on 1 site. My business took a significant sales hit starting early July 2013, I know google did and algorithm update that did have SEO aspects. I need to resolve the problem so I can stay in business
-
Thx Jane- No I wasn't aware of that. I don't get it because I put canonical tags right under the Head and I used the code below to do it. I will check again but am unsure how to fix it
I don't even know how to fix coding on the Http://cheaptubes.com site. It seems like when I add content to the canonical site it updates all of them. Thx for pointing out errors, you are giving me something to fix and improve.
-
Hi again,
Are you aware that you have a canonical tag on http://cheaptubes.com that points to a non-existent URL? i.e. http://i.imgur.com/yEd2377.png
http://www.cheaptubes.com/default.html
If http://cheaptubes.com/ 301 redirected to http://www.cheaptubes.com/, this would resolve the issue.
Are you aware that the www version of your site shows for a brand search (https://www.google.co.uk/search?q=cheaptubes.com&oq=cheaptubes.com&aqs=chrome..69i58j69i60l2j69i57j69i60j0.3367j0j4&sourceid=chrome&es_sm=91&ie=UTF-8) but that the canonical tags on each page point to the non-www version, e.g. http://i.imgur.com/P7Tizsv.png and http://i.imgur.com/lhTA95w.png?
The canonical tag on the www.cheaptubes.com/ page also points to http://www.cheaptubes.com/default.html. Sorry to show so many errors, but it doesn't look like canonicalisation has been implemented properly here.
-
Thx Jane - I may not have put a canonical tag on that page yet but its the same for every page. I can't access the http://cheaptubes.com but I can access the canonical version to publish. I did put canonical tags on most of my other pages such as the SWNTs page but it still shows a non canonical version when moz crawls it. Perhaps a 301 from http://cheaptubes.com to the canonical page? I'm just not sure how to handle it.
-
Hi Mike,
Are you saying that there is a canonical tag on http://cheaptubes.com/cntmaterialsafetydatasheet.htm, pointing to http://www.cheaptubes.com/cntmaterialsafetydatasheet.htm? This would solve the duplicate content problem, but I do not see a canonical tag on either of those pages...
-
Hi Everyone - I'm hoping you can help me out again. I have a functional 301 on cheaptubesinc.com. that cleared about 1/2 my dup content penalty on the moz crawl this week. As you can see in the results below, I still have 57 pages with dup content according to Moz.
57 Duplicate Page Content
13 4XX (Client Error)
57 Duplicate Page TitleI checked and I think it is mostly a canonical problem. I do have Rel Canonical tags on all my pages. When I clicked on the 1st one it appears that is the case, see below
cheaptubes.com carbon nanotubes msds
http://www.cheaptubes.com/cntmaterialsafetydatasheet.htm29414872001 duplicate
cheaptubes.com carbon nanotubes msds
http://cheaptubes.com/cntmaterialsafetydatasheet.htm25My question is, do I need another 301 from http://cheaptubes.com to the canonical version? I'ld rather not since I had to fight with network solutions for a week for them to add the / after .com so my other pages would work. Is this a penalty I should still be concernedabout given that I have the rel canonical tags? Please let me know your thoughts on thisMike
-
Thanks Everyone. I got the old url cheaptubesinc.com 301'd to cheaptubes.com this week. Of course network solutions left off the / after .com and before the page name so that only the home page would 301 and they could try to sell me more 301s, it cost $60 for 1 and I have 48 pages on my site. I called and emailed them all week and they kept saying they had done it right and and they couldn't force google to change the links. I then realized if I typed www.cheaptubesinc/graphene.htm that it didn't work because it 301'd to www.cheaptubes.comgraphene.htm. They were argumentative with me even though I was polite with them even though I didn't want to be. I finally got a tech on the phone who said he would add the slash and ask his boss for forgiveness. However given the history of having the domain parked and pointed before and that not working over time & now this, I think my best bet is to transfer my domains to someone else. I heard bluehost is good. My concern is if they were that unethical in our dealings and the boss was argumentative in emails than they could go in an remove the slash at any time.
I also found a ton of code errors right at the top of my pages. I now know it was from putting up temporary messages but not checking to make sure the code was clean. The woman I bought my them from (6.5 years ago I paid her $60 and she still helps me for free, what difference between her & NS) notice open H1's & P elements at the top of the pages. I was still ranking well for acronyms but missing out on the long keywords since last july which caused my sales to drop off. I figure I lost at least 150K in sales because I neglected my website and didn't clean up the code on my pages a painful lesson I won't soon forget. On tuesday, when I searched single walled carbon nanotubes I had to go 8 pages back in google to find my page. By week's end I was #8 on page 1 and ahead of sigma aldrich a major materials supplier.
Thank you so much for your help everyone, it is sincerely appreciated
Mike
-
Thank you Oleg - I did put the tag
into the head right below the robots & google bot code on every page. I mistakenly deleted some very old non updated pages. Thx to moz, i have a list of the pages and will contact hosting co to 301 it. I think I ultimately have to 301 each page. I had moz recrawl my site last night but it said it dropped from 100 duplicate content penalties to 89, an improvement but not the one I hoped for. I did have a client tell my the site was down today, contacted network solutions and they said it was up now but they had an outage last night. Perhaps it affected the moz recrawl,but I can't know that. I also want to change the names of the pages as an interim measure before I update the site to newer format. Should I create new optimized by name pages first and then get on the phone with tech support and 301 them all to the knew pages? seems logical but so did deleting old pages until moz couldn't find them, then i realized the bots will count it against me rather than the housekeeping that it was.
Mike
Mike
-
Read these two posts... they cover everything.
-
ok, got it, thank you so much Jane
-
Hi,
You don't need to redirect at all (with a 301 or otherwise) if the canonical tag is in place. So don't worry about that at all - both URLs can load together if the canonical tag points Google from the "duplicate" to the "correct / canonical" one. Sorry if that wasn't clear.
I am not sure the frequency of Moz's crawling or if you can force a refresh, I'm sorry.
-
Thx Jane
The problem is I can't simply 301 it because I'm not on apache. I can do the canonical tag. Of course I've already gone in and changed it over to the tag + refresh but server is down so it won't publish right now. I was trying to get it done ahead of moz crawling my site today. Is there a way to get moz to recrawl it after the changes are updated or do I need to wait another week?
-
Hi,
Hard to say, but it definitely won't have helped. As Bryan says, you've split authority between over twice the number of pages the site should have, and Google can take action against sites that produce a large amount of duplicate content. I'd get the canonical tags in place (and thoroughly check they're set up right, as it can be a mess if they're implemented incorrectly) and check on progress over two or three weeks. If you see nothing happen, I'd say your reason for dropping could be something else.
-
Hi again,
The canonical tag sounds like the right way to go for you.
Regarding the meta refresh method of redirection - this works perfectly for users... it was always the case that search engines did not honour this as a redirect though. This may have changed in the recent past (and realistically, it should have - a lot of people used this tactic for redirection and Google should understand that it shows a moved page). However, it is generally thought that the meta refresh does not pass all authority (as noted here), and this thread shows a Googler advising against it (this is a post from 2010 though).
Honestly, with the canonical tag, you don't need to do the refresh / redirection - this will take care of the issue
Cheers,
Jane
-
Hi there,
I'll answer these one at a time as there are a few responses to go through.
default.htm is the home page as created by the CMS, but you want to either use that URL or www.cheaptubes.com as the home page, not both.
The solution is a 301 or the canonical tag so that home page content does not appear on both URLs.
-
Hi Jane, Oleg, & Bryan
I checked with the woman who designed my theme (she is awesome). She offered the following suggestions which seem like the way to go for me. Are there any negatives that I'm not aware of with the options below?
Since you are still using FrontPage, just open your site, locate the appropriate pages, and type the following into the head area:
If you are on a Windows server, your web host can do the 301 redirect for you. You will tell them the name of the old pages and the name of the new pages and they will do the rest.
An easy alternate is for you to do the redirect yourself with an easy tag that goes into the head area of the old pages. This tag is called a redirect and redirects from the old page to the new one.
URL="http://www.newsite.com/newurl.html">
Google, Bing, and Yahoo all recognize the meta tag for the redirect and will adjust their indexing accordingly. I will usually leave an old page on the server for about 3 months to give the search engines time to catch up. Then I can delete the page.
You can, of course, get more "bang for your buck" by using both the canonical link and the meta refresh at the same time.
URL="http://www.newsite.com/newurl.html">
I like the last one, am going to try that unless you think its a flawed strategy.
Thanks for your help
Mike
-
Hi Jane
How do I change to canonical url's if I can't do a 301?
Mike
-
so how do I use the canonical tag since i can't 301 it?
-
It certainly could. Google sees the www. version as a 2nd website, so essentially you're splitting your 'ranking authority' between 2 webpages.
-
Thanks Oleg
I can't 301 because I'm not using apache, still on frontpage. I know its old, getting out my abacus now : )
-
To sum up...
- 301 redirect all non-www urls to www versions (since it has a higher page authority) and add canonicals to all pages with the www version of the url
- For all lower case / upper case page duplicates... pick one, set a canonical tag and 301 to the chosen case, make sure all your links point to the correct url case.
- 301 redirect default.htm to your root domain - http://www.cheaptubes.com
-
does the 2 versions problem help to explain why my sales started dropping significantly after the google july 4th update? I know there were some SEO penalties in that update. I also know a friendly competitor who saw a similar drop starting in early July.
-
Hi Jane
Thank you so much. I am reviewing the link you provided. I don't think I can 301 redirect because it is done in front page, not apache. I have tried for years to find another platform but failed. I spent years trying to figure out drupal, even ordered several books but no luck. I tried concrete 5 and just using HTML 5 editor like coffee cup. I keep struggling with getting them to work. I've bought themes to use but can't get them operational.
I thought default.htm was supposed to be the home page, is that incorrect?
Mike
-
Hi again,
Yep - your non-www and www pages are both resolving... e.g. http://cheaptubes.com/ and http://www.cheaptubes.com/ bring up the same content. Also, http://cheaptubes.com/default.htm and http://www.cheaptubes.com/default.htm is also a duplicate of the home page.
Internally, I am seeing the same thing, e.g. http://www.cheaptubes.com/carbon-nanotubes-prices.htm and http://cheaptubes.com/carbon-nanotubes-prices.htm - same page, one on the www subdomain ("www." is a subdomain like any other, just with an extremely common name) and one just sitting on the root.
The solution here is either to 301 redirect the non-www version of the site to the www version for every page, or to use the canonical tag to point from the non-preferred versions to the "canonical" versions. More information on this is available here.
You also have a situation where upper-case URLs will resolve as well as lower case ones, e.g. http://www.cheaptubes.com/SWNTs.htm and http://www.cheaptubes.com/swnts.htm (as well as http://cheaptubes.com/swnts.htm!).
URLs should only be allowed to resolve with one case, preferably lower. The upper / mixed case should 301 redirect to the proper version.
Essentially, the "two versions of the site" issue is the biggest problem, with all pages being available on at least two URLs - one with www and one without. There are other tidiness issues like /default.htm bringing up the home page as well.
Does this make sense? Let me know if this is not clear.
Best,
Jane
-
also on a page that moz ranks as an "F", I still rank high in organic results, see the results from when I searched for MWNTs below, I was 1st organic result. If long form, multi walled carbon nanotubes I fell to 6th or 7th but still on the first page.
-
Thank you Oleg, Bryan, & Jane. I am a rookie when it comes to web development but my pages always ranked well because enough Moz tips sunk in. See the alert from last weeks crawl below.
Pages with High Priority Issues
98Duplicate Page Content24XX (Client Error)If we look at the home page, it has 3 URLs, see belowURLPage AuthorityLinking Root DomainsExternal Link CountInternal Link CountStatus CodeDuplicate URLsDownload Duplicates
cheaptubes.com the source for carbon nanotubes home page
http://www.cheaptubes.com3322611882003 duplicates
cheaptubes.com the source for carbon nanotubes home page
http://www.cheaptubes.com/default.htm2422622001 of 3 duplicates
cheaptubes.com the source for carbon nanotubes home page
http://cheaptubes.com2931502002 of 3 duplicates
cheaptubes.com the source for carbon nanotubes home page
http://cheaptubes.com/default.htm2410462003 of 3 duplicates Does this help?-Mike
-
Hi there,
This could definitely be a case of both non-www and www URLs resolving, but I'd like to echo the guys above me and ask for more information - if you could share the actual examples, either on here or in a private message, it would be easier to find why Moz has found twice the number of URLs your site should have.
Thanks,
Jane
-
+Really need more information
If you have URLs constructed dynamically depending on where the user navigates from this could also be an issue, but I would expect more than 2x the pages.
-
Could you share more details?
What do the duplicate content examples look like? http vs https? www. vs non-www?
If the content is replicated on 2 domains, yes that is duplicate content and you should consolidate to one site via 301 redirects.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Products description from third party vendor creating duplicate content issues?
Hi, I am running my client's e-store. The store sells different products from various vendors. Vendors provide us product descriptions. The problem is that these vendors also give these description to display their products on other similar sites and hence creating duplicate content issue. Thanks.
On-Page Optimization | | Kashif-Amin0 -
Duplicate product content/disclaimers for non-e-commerce sites
This is more a follow-up to Rand's recent Whiteboard "Handling User-Generated & Manufacturer-Required Duplicate Content Across Large Numbers of URLs." I posed my question in the comments, but unsure it will get picked up. My situation isn't exactly the same, but it's similar: Our site isn't an e-commerce site and doesn't have user reviews yet, but we do have maybe 8 pages across 2 product categories featuring very similar product features with duplicate verbiage. However, we don't want to re-write it because we want to make it easy for users to compare apples-to-apples to easily see which features are actually different. We also have to run disclaimers at the bottom of each page.\ Would i-framing the product descriptions and disclaimers be beneficial in this scenario, with the addition of good content? It would still be nice to have some crawlable content on those pages, so the i-framing makes me nervous unless we compensate with at least some above-the-fold, useful content that could be indexed. Thanks, Sarah
On-Page Optimization | | sbs2190 -
Would This Be Highlighted as Duplicate Meta Desc?
Due to the nature of our business we literally have hundreds of web pages and so we recently asked our Web Developers to create the Meta Description for many of the pages that they create - unfortunately they don't seem to understand SEO but before I go back to them I wanted to run this by the MOZ Community... Here is a snippet of the Meta Desc they have created, personally I'm not happy with it BUT I need to be certain on one thing... Will Google determine that it is (almost) duplicated Meta content? Why not visit Cape Town on an Africa Cruise with XYZ Cruises? Book with us today for great cruise deals, special offers and exclusive club benefits. Why not visit Casablanca on an Africa Cruise with XYZ Cruises? Book today for great cruise deals, special offers and exclusive club benefits. Why not visit Durban on an Africa Cruise with XYZ Cruises? Book with us today for great cruise deals, special offers and exclusive club benefits. Why not visit Port Elizabeth on an Africa Cruise with XYZ Cruises? Book today for great cruise deals, special offers and exclusive club benefits. Why not visit Aukland on an Australasia Cruise with XYZ Cruises? Book today for great cruise deals, special offers and exclusive club benefits. Why not visit Christchurch on an Australasia Cruise with XYZ Cruises? Book today for great cruise deals, special offers and exclusive club benefits. Why not visit Melbourne on an Australasia Cruise with XYZ Cruises? Book today for great cruise deals, special offers and exclusive club benefits. Why not visit Sydney on an Australasia Cruise with XYZ Cruises? Book today for great cruise deals, special offers and exclusive club benefits. Why not visit Orlando on a Caribbean Cruise with XYZ Cruises? Book with us today for great cruise deals, special offers and exclusive club benefits. Why not visit Central America Cruise on a Cruise Destinations with XYZ Cruises? Book today for great cruise deals, benefits and special offers. Why not visit Acapulco on a Central America Cruise with XYZ Cruises? Book today for great cruise deals, special offers and exclusive club benefits. Why not visit Puntarenas on a Central America Cruise with XYZ Cruises? Book today for great cruise deals, special offers and exclusive club benefits. Why not visit Sharm El Sheik on an Egypt And Red Sea with XYZ Cruises? Book today for great cruise deals, special offers and exclusive club benefits. Why not take a European Cruise with XYZ Cruises? Book with us today for great cruise deals, special offers and exclusive cruise club benefits. Why not visit Bangkok on a Far East & Indian Ocean Cruise with XYZ Cruises? Book today for great cruise deals, special offers and exclusive benefits. Why not visit Halong Bay on a Far East & Indian Ocean Cruise with XYZ Cruises? Book today for great cruise deals, offers and exclusive benefits. Why not visit Hong Kong on a Far East & Indian Ocean Cruise with XYZ Cruises? Book today for great cruise deals, special offers and exclusive benefits. Why not visit Kuala Lumpur on a Far East & Indian Ocean Cruise with XYZ Cruises? Book today for great cruise deals, offers and exclusive benefits. Why not visit Okinawa on a Far East & Indian Ocean Cruise with XYZ Cruises? Book today for great cruise deals, offers and exclusive benefits. Why not visit Phuket on a Far East & Indian Ocean Cruise with XYZ Cruises? Book today for great cruise deals, special offers and exclusive benefits. Why not visit Shanghai on a Far East & Indian Ocean Cruise with XYZ Cruises? Book today for great cruise deals, offers and exclusive benefits. Why not visit Singapore on a Far East & Indian Ocean Cruise with XYZ Cruises? Book today for great cruise deals, offers and exclusive benefits. Why not visit Tokyo on a Far East & Indian Ocean Cruise with XYZ Cruises? Book today for great cruise deals, special offers and exclusive benefits. Like I said earlier, there are hundreds of pages more! I think that if I want total uniqueness then it's going to be a laborious task of doing them all individually but I want to make sure that it is worth it?! Thanks Andy
On-Page Optimization | | TomKing0 -
Duplicate Home Page
Hi, I have a question around best practise on duplicate home pages. The /index.aspx page is showing up as a top referrer in my analytics. I have the rel=canonical tag implemented for the www.mysite.com on both pages. Do I need to 301 the /index.aspx to the mysite.com? I have a lot of links pointing to the /index.aspx (half of those are coming from the mysite.com). www.mysite.com/index.aspx www.mysite.com Many thanks Jon
On-Page Optimization | | JonRaubenheimer0 -
New Client Wants to Keep Duplicate Content Targeting Different Cities
We've got a new client who has about 300 pages on their website that are the same except the cities that are being targeted. Thus far the website has not been affected by penguin or panda updates, and the client wants to keep the pages because they are bringing in a lot of traffic for those cities. We are concerned about duplicate content penalties; do you think we should get rid of these pages or keep them?
On-Page Optimization | | waqid0 -
Duplicate Content - Delete it or NoIndex?
Last month I realized that one of my freelancers had been feeding my website with copied / spun content and sadly, there's lots of it. And of course it got my website to be hit hard by the last Panda update. Now that I've identified the content, what the best thing to do? Should I delete it permanently and get 404 errors or should I set the pages' robot meta tag to "nofollow"?
On-Page Optimization | | sbrault740 -
Should I worry about duplicate titles on pages where there is paginated content?
LivingThere.com is a real estate search site and many of our content pages are "search result" - ish in that a page often provides all the listings that are available and this may go on for multiple pages. For example, this is a primary page about a building: http://livingthere.com/building/31308-Cocoa-Exchange Because of the number of listings, the listings paginate to a second page: http://livingthere.com/building/31308-Cocoa-Exchange?MListings_page=2 Both pages have the same Page Title. Is this a concern? If so is there a "best practice" for giving paginated content different titles? Thanks! Nate
On-Page Optimization | | nate1230 -
Cross Domain Duplicate Content
Hi My client has a series of websies, one main website and several mini websites, articles are created and published daily and weekly, one will go on a the main website and the others on one, two, or three of the mini sites. To combat duplication, i only ever allow one article to be indexed (apply noindex to articles that i don't wanted indexed by google, so, if 3 sites have same article, 2 sites will have noindex tag added to head). I am not completely sure if this is ok, and whether there are any negative affects, apart from the articles tagged as noindex not being indexed. Are there any obvious issues? I am aware of the canonical link rel tag, and know that this can be used on the same domain, but can it be used cross domain, in place of the noindex tag? If so, is it exactly the same in structure as the 'same domain' canonical link rel tag? Thanks Matt
On-Page Optimization | | mattys0