per google webmaster tools:

If Google knows that these pages have the same content, we may index only one version for our search results. Our algorithms select the page we think best answers the user's query. Now, however, users can specify a canonical page to search engines by adding a element with the attribute rel="canonical" to the section of the non-canonical version of the page. Adding this link and attribute lets site owners identify sets of identical content and suggest to Google: "Of all these pages with identical content, this page is the most useful. Please prioritize it in search results."

Well, the answer would be yes and no. A robots.txt file would stop the bots from indexing the page, but links from other pages in site to that non indexed page could therefor make it crawlable and then indexed. AS posted in google webmaster tools here:

"You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one).

While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project (www.dmoz.org), can appear in Google search results."

I think the best way to avoid any conflict is applying the rel="canonical"  tag to each duplicate page that you don't want indexed.

You can find more info on rel canonical here

Hope this helps out some.

1. Yes, the link juice will eventually be applied to the new page. This is part of the benefit behind 301 redirect.It sometimes just takes a little time for it to process and show up in reports.

2. The link juice or link equity will eventually pass through to new url once Google has established and indexed the new page. This could take weeks to months.

3. I would continue to optimized on a regular basis. Make sure that after each 301 redirect you also adjust all internal linking to the new page, especially in your sitemaps! Other wise it could get confusing for both visitors and bots.

Hope this helps.

Have you updated all internal links from old to new shortened title page? Double check that first then resubmit sitemap to see if you can get re-indexed faster.

Thank you for the feedback and help.

I have looked up url removal in webmaster tools and it states that the page must be removed from the site. If I remove index.html I wont have a home page. Am I understanding you correctly? Heres what google states on url removal.

To remove a page or image, you must do one of the following:

• Make sure the content is no longer live on the web. Requests for the page must return an HTTP 404 (not found) or 410 status code.
• Block the content using a robots.txt file.
• Block the content using a meta noindex tag.

Please clarify when you get a moment.

I would have thought the htaccess 301 redirects from www.domain.com/index.html to www.domain.com would be enough.

Thank you in advance.

HTTP/1.1 301 Moved Permanently => Date => Tue, 08 May 2012 13:44:26 GMT Server => Apache/2.0.52 (CentOS) Location => http://www.domain.com/ Content-Length => 330 Connection => close Content-Type => text/html; charset=iso-8859-1

I have a site that was recently hit by the Google penguin update and dropped a page back. When running the site through seomoz tools, I keep getting duplicate content in the reports for domain.com and domain.com/index.html, even though I have a 301 rewrite condition. When I test the site, domain.com/index.html redirects to domain.com for all directories and root. I don't understand how my index page can still get flagged as duplicate content.

I also have a redirect from domain.com to www.domain.com.

Is there anything else I need to do or add to my htaccess file?

Appreciate any clarification on this.