Domain.com and domain.com/index.html duplicate content in reports even with rewrite on

[anthonytjm]

I have a site that was recently hit by the Google penguin update and dropped a page back. When running the site through seomoz tools, I keep getting duplicate content in the reports for domain.com and domain.com/index.html, even though I have a 301 rewrite condition. When I test the site, domain.com/index.html redirects to domain.com for all directories and root. I don't understand how my index page can still get flagged as duplicate content.

I also have a redirect from domain.com to www.domain.com.

Is there anything else I need to do or add to my htaccess file?

Appreciate any clarification on this.

[SeanKoenig]

Hello Anthony,

Saw this still open.

If your index.html "Rewrite" code is accurate, could the issue be WWW, i.e. http://www.domain.com vs. http://domain.com?

RewriteCond %{HTTP_HOST} ^domain.com
RewriteRule ^(.*)$ http://www.domain.com/$1 [R=permanent,L]

[Dr-Pete]

I checked one of your campaigns, and it does seem like the 301-redirect is working properly. I'm also not seeing any evidence of links to the "index.htm" version or other issues. I don't see evidence of both version sin Google's index. Not sure exactly what's going on here, but I'll run it by the support team. I don't think you have cause for concern.

[anthonytjm]

Thank you for the feedback and help.

I have looked up url removal in webmaster tools and it states that the page must be removed from the site. If I remove index.html I wont have a home page. Am I understanding you correctly? Heres what google states on url removal.

To remove a page or image, you must do one of the following:

• Make sure the content is no longer live on the web. Requests for the page must return an HTTP 404 (not found) or 410 status code.
• Block the content using a robots.txt file.
• Block the content using a meta noindex tag.

Please clarify when you get a moment.

I would have thought the htaccess 301 redirects from www.domain.com/index.html to www.domain.com would be enough.

Thank you in advance.

[irvingw]

a) request removal of the /index.html URL in webmaster tools and it will go away in Google's index quickly.

b) make sure that when you link to your homepage on your site you are not linking to the /index.html URL -  I bet you are somewhere    do a sitewide search in dreamweaver to find all instances and do a global replace.

[AJPro]

It could take a little time. I did some redirects myself earlier this year, but the old pages are still in Google's index.

Maybe someone else can confirm that it can take a little time before the old pages are dropped from Google's index?

[anthonytjm]

HTTP/1.1 301 Moved Permanently => Date => Tue, 08 May 2012 13:44:26 GMT Server => Apache/2.0.52 (CentOS) Location => http://www.domain.com/ Content-Length => 330 Connection => close Content-Type => text/html; charset=iso-8859-1

[AJPro]

Did you verify with a tool like http://www.webconfs.com/http-header-check.php that you get a 301 redirect?