Domain.com and domain.com/index.html duplicate content in reports even with rewrite on
-
I have a site that was recently hit by the Google penguin update and dropped a page back. When running the site through seomoz tools, I keep getting duplicate content in the reports for domain.com and domain.com/index.html, even though I have a 301 rewrite condition. When I test the site, domain.com/index.html redirects to domain.com for all directories and root. I don't understand how my index page can still get flagged as duplicate content.
I also have a redirect from domain.com to www.domain.com.
Is there anything else I need to do or add to my htaccess file?
Appreciate any clarification on this.
-
Hello Anthony,
Saw this still open.
If your index.html "Rewrite" code is accurate, could the issue be WWW, i.e. http://www.domain.com vs. http://domain.com?
RewriteCond %{HTTP_HOST} ^domain.com
RewriteRule ^(.*)$ http://www.domain.com/$1 [R=permanent,L] -
I checked one of your campaigns, and it does seem like the 301-redirect is working properly. I'm also not seeing any evidence of links to the "index.htm" version or other issues. I don't see evidence of both version sin Google's index. Not sure exactly what's going on here, but I'll run it by the support team. I don't think you have cause for concern.
-
Thank you for the feedback and help.
I have looked up url removal in webmaster tools and it states that the page must be removed from the site. If I remove index.html I wont have a home page. Am I understanding you correctly? Heres what google states on url removal.
To remove a page or image, you must do one of the following:
- Make sure the content is no longer live on the web. Requests for the page must return an HTTP 404 (not found) or 410 status code.
- Block the content using a robots.txt file.
- Block the content using a meta noindex tag.
Please clarify when you get a moment.
I would have thought the htaccess 301 redirects from www.domain.com/index.html to www.domain.com would be enough.
Thank you in advance.
-
a) request removal of the /index.html URL in webmaster tools and it will go away in Google's index quickly.
b) make sure that when you link to your homepage on your site you are not linking to the /index.html URL - I bet you are somewhere do a sitewide search in dreamweaver to find all instances and do a global replace.
-
It could take a little time. I did some redirects myself earlier this year, but the old pages are still in Google's index.
Maybe someone else can confirm that it can take a little time before the old pages are dropped from Google's index?
-
HTTP/1.1 301 Moved Permanently => Date => Tue, 08 May 2012 13:44:26 GMT Server => Apache/2.0.52 (CentOS) Location => http://www.domain.com/ Content-Length => 330 Connection => close Content-Type => text/html; charset=iso-8859-1
-
Did you verify with a tool like http://www.webconfs.com/http-header-check.php that you get a 301 redirect?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate Content/Missing Meta Description | Pages DO NOT EXISIT!
Hello all, For the last few months, Moz has been showing us that our site has roughly 2,000 duplicate content errors. Pages that were actually duplicate content, I took care of accordingly using best practice (301 redirects, canonicalization,etc.). Still remaining after these fixes were errors showing for pages that we have never created. Our homepage is www.primepay.com. An example of pages that are being shown as duplicate content is http://primepay.com/blog/%5BLink%20to%20-%20http:/www.primepay.com/en/payrollservices/payroll/payroll/payroll/online-payroll with a referring page of http://primepay.com/blog/%5BLink%20to%20-%20http:/www.primepay.com/en/payrollservices/payroll/payroll/online-payroll. Some of these are even now showing up as 403 and 404 errors. The only real page on our site within that URL strand is primepay.com/payroll or primepay.com/payroll/online-payroll. Therefore, I am not sure where Moz is getting these pages from. Another issue we are having in relation to duplicate content is that moz is showing old campaign url’s tacked on to our blog page i.e. http://primepay.com/blog?title=&page=2&utm_source=blog&utm_medium=blogCTA&utm_campaign=IRSblogpost&qt-blog_tabs=1. As of this morning, our duplicate content went from 2,000 to 18,000. I exported all of our crawl diagnostics data and looked to see what the referring pages were, and even they are not pages that we have created. When you click on these links, they take you to a random point in time from the homepage of our blog; some dating back to 2010. I checked our crawl stats in both Google and Bing’s Webmaster tool, and there are no duplicate content or 400 level errors being reporting from their crawl. My team is truly at a loss with trying to resolve this issue and any help with this matter would be greatly appreciated.
Moz Pro | | PrimePay0 -
Duplicate Content & Rel Canonical Tag not working
I'm really questioning the legitimacy of the duplicate content flags with moz. I'm building a website that sells home decor products and a lot of the pages are similar in structure (As would be expected with a store that sells thousands of individual products). It seems a little overkill to me to flag the following pages as duplicate content. They have different urls, titles, h1, h2, and h3 tages, different meta tags, etc. Right now, it's saying that the following have duplicate page content: http://www.countryporchhomedecor.com
Moz Pro | | cp_web
http://countryporchhomedecor.com/park-designs/pillows/christmas-vacation-embroidered-pillow
http://www.countryporchhomedecor.com/donna-sharp/throws/camo-bear-throw
http://countryporchhomedecor.com/park-designs/teapots/wonderland-teapot
http://countryporchhomedecor.com/park-designs/rag-rugs/cambridge-rug-36x60
http://www.countryporchhomedecor.com/donna-sharp/lodge-quilts/king%2C-woodland
http://www.countryporchhomedecor.com/park-designs/rag-rugs/redmon-rag-rug-36x60
http://www.countryporchhomedecor.com/park-designs/valances/hearthside-valance-72x14
http://countryporchhomedecor.com/park-designs/valances/hearthside-valance-72x14
http://countryporchhomedecor.com/donna-sharp/lodge-quilts/king,-woodland
http://www.countryporchhomedecor.com/park-designs/teapots/wonderland-teapot
http://countryporchhomedecor.com/donna-sharp/throws/camo-bear-throw
http://countryporchhomedecor.com/park-designs/accessories/home-place-tumbler
http://www.countryporchhomedecor.com/donna-sharp/lodge-quilts/king,-woodland
http://www.countryporchhomedecor.com/park-designs/rag-rugs/cambridge-rug-36x60
http://www.countryporchhomedecor.com/park-designs/pillows/christmas-vacation-embroidered-pillow
http://www.countryporchhomedecor.com/donna-sharp/lodge-quilts/king%2C-woodland?pi=18
http://countryporchhomedecor.com/donna-sharp/lodge-quilts/king%2C-woodland
http://www.countryporchhomedecor.com/park-designs/accessories/home-place-tumbler
http://countryporchhomedecor.com/park-designs/rag-rugs/redmon-rag-rug-36x6 Any ideas? Also, it seems like it's not honoring the rel-canonical tag. It keeps saying that pages with a rel canonical tag are duplicates when some of the urls that it's flagging shouldn't even be indexed because of the canonical tag. The "pi" in the query string should not be indexed! http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?pi=18&page=3
http://countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=6
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=7
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?pi=18&page=6
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=10
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?pi=18&page=8
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=8
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams
http://countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=7
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?pi=18&page=7
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=1
http://countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=8
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?pi=18&page=5
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?pi=18&page=10
http://countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=3
http://countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=5
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?pi=18&page=4
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?pi=18&page=9
http://www.countryporchhomedecor.com/bedding-%26-quilts/shams/standard-shams
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?pi=18
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?pi=18&page=1
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=6
http://countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=1
http://countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=5
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=2
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=9
http://countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=4
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=3
http://countryporchhomedecor.com/bedding-%26-quilts/shams/standard-shams
http://www.countryporchhomedecor.com/bedding-%26-quilts/shams/standard-shams?pi=18
http://countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=9
http://countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=10
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?pi=18&page=2
http://countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=2
http://www.countryporchhomedecor.com/bedding-&-quilts/shams/standard-shams?page=40 -
In alt tag of a image can we use #hashtag or domain.com ? Is that good SEO or not allowed ?
Some of the Google Search shows a title has a hashtag of an article, which contain keyword and while tweeting them, the title which has a hashtag automatically very good used for getting traffic to the blog. And other one, can we use the hash tag inside the alt attribute ? Or our domain name with .com in it. Like Google.com or #Google ?
Moz Pro | | Esaky0 -
SEOTools Reporting
Is there any other services or software that provide keyword ranking and traffic data other than SEOMoz or Raven Tools?
Moz Pro | | JohnW-UK0 -
Where is the labs LDA topics tool? The articles mentioning it point to http://www.seomoz.org/labs/lda but that forwards me to the tools landing page.
I searched SEOMOZ for this but haven't yet found the answer. It seems that it's been removed. I'm sure it was announced somewhere that I missed
Moz Pro | | bluenote0 -
Blogger Duplicate Content? and Canonical Tag
Hello: I previously asked this question, but I would love to get more perspectives on this issue. In Blogger, there is an archive page and label(s) page(s) created for each main post. Firstly, does Google, esp. considering Blogger is their product, possibly see the archive and tag pages created in addition to the main post as partial duplicate content? The other dilemma is that each of these instances - main post, archive, label(s) - claim to be the canonical. Does anyone have any insight or experience with this issue and Blogger and how Google is treating the partial duplicates and the canonical claims to the same content (even though the archives and label pages are partial?) I do not see anything in Blogger settings that allows altering these settings - in fact, the only choices in Blogger settings are 'Email Posting' and 'Permissions' (could it be that I cannot see the other setting options because I am a guest and not the blog owner?) Thanks so much everyone! PS - I was not able to add the blog as a campaign in SEOmoz Pro, which in and of itself is odd - and which I've never seen before - could this be part of the issue? Are Blogger free blogs not able to be crawled for some reason via SEOmoz Pro?
Moz Pro | | holdtheonion0 -
Page and Domain Authority and other bits
Hi, I am in the process of finding blogs to have a few articles published with a couple of links in each. Articles will all be unique and relevant to the link I drop in and relevant in someway to the reader However I have a few questions. My site is a designer menswear site, so I have picked fashion and sports sites first and foremost to have the articles published. Now, I have a guy who owns about 30 different websites. 2 of them are sports based and about 10 are fashion based. Around $10-$15 an article. I have ran them all through the Open Site Explorer Tool and picked out the best ranked ones. Now my problem is, how do I know if its a good site to not only list an article on, but to pay for it as well. The sites page ranks are around the 30-45 range, the domains are around the 35-45 range. What is a good range to have? I know the higher the better but is 30-45 good enough to pay for? (I don't mind paying the $10 each (£7 my money) for each one) Also as he is quoted me in dollars, I assume there all USA based, so majority of users are USA based. Well I am UK based and only ship to the UK. Will this matter as much if I am trying to gain backlinks? Obviously a UK based site, would be ideal, but is it a case of getting more external links on the web for Google to find, as long as they are relevant to the user? Any help would be great. Thanks Will
Moz Pro | | WillBlackburn0