Domain.com and domain.com/index.html duplicate content in reports even with rewrite on
-
I have a site that was recently hit by the Google penguin update and dropped a page back. When running the site through seomoz tools, I keep getting duplicate content in the reports for domain.com and domain.com/index.html, even though I have a 301 rewrite condition. When I test the site, domain.com/index.html redirects to domain.com for all directories and root. I don't understand how my index page can still get flagged as duplicate content.
I also have a redirect from domain.com to www.domain.com.
Is there anything else I need to do or add to my htaccess file?
Appreciate any clarification on this.
-
Hello Anthony,
Saw this still open.
If your index.html "Rewrite" code is accurate, could the issue be WWW, i.e. http://www.domain.com vs. http://domain.com?
RewriteCond %{HTTP_HOST} ^domain.com
RewriteRule ^(.*)$ http://www.domain.com/$1 [R=permanent,L] -
I checked one of your campaigns, and it does seem like the 301-redirect is working properly. I'm also not seeing any evidence of links to the "index.htm" version or other issues. I don't see evidence of both version sin Google's index. Not sure exactly what's going on here, but I'll run it by the support team. I don't think you have cause for concern.
-
Thank you for the feedback and help.
I have looked up url removal in webmaster tools and it states that the page must be removed from the site. If I remove index.html I wont have a home page. Am I understanding you correctly? Heres what google states on url removal.
To remove a page or image, you must do one of the following:
- Make sure the content is no longer live on the web. Requests for the page must return an HTTP 404 (not found) or 410 status code.
- Block the content using a robots.txt file.
- Block the content using a meta noindex tag.
Please clarify when you get a moment.
I would have thought the htaccess 301 redirects from www.domain.com/index.html to www.domain.com would be enough.
Thank you in advance.
-
a) request removal of the /index.html URL in webmaster tools and it will go away in Google's index quickly.
b) make sure that when you link to your homepage on your site you are not linking to the /index.html URL - I bet you are somewhere do a sitewide search in dreamweaver to find all instances and do a global replace.
-
It could take a little time. I did some redirects myself earlier this year, but the old pages are still in Google's index.
Maybe someone else can confirm that it can take a little time before the old pages are dropped from Google's index?
-
HTTP/1.1 301 Moved Permanently => Date => Tue, 08 May 2012 13:44:26 GMT Server => Apache/2.0.52 (CentOS) Location => http://www.domain.com/ Content-Length => 330 Connection => close Content-Type => text/html; charset=iso-8859-1
-
Did you verify with a tool like http://www.webconfs.com/http-header-check.php that you get a 301 redirect?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is MOZ crawl is returning URLs with variable results showing Missing Meta Desc? Example: http://nw-naturals.net/?page_number_0=47
Can you help me dive down into my website guts to find out why the MOZ crawl is returning URLs with variable results? And saying this is missing a description when it's not really a page? Example: http://nw-naturals.net/?page_number_0=47. I've asked MOZ but it's a web development issue so they can't help me with it. Has anyone had an issue with this on their website? Thank you!
Moz Pro | | lewisdesign0 -
Duplicate content - Product Categories
Dears, I've use "Site Crawl" tool to find any SEO warnings, and I found 991 duplicated content. The problem is that the pages are not duplicated its all products category pages, please check this exmaple: This page: https://www.jobedu.com/en/shop/category/prints/Postcards/na/all-colors/all-size and this page: https://www.jobedu.com/en/shop/category/Accessories/keychain/na/all-colors/all-size It said its duplicated, and it's 991 pages! How to fix this this? what I can do?
Moz Pro | | jobedu0 -
Duplicate content on SearchResults.asp
hi guys. I'm currently working through the reported crawl errors in Moz Analytics, but an unsure what to do about some of them. for example... Searchresults.asp?search=frankie+says+relax is showing as having duplicate page content and page title as SearchResults.asp?searching=Y&sort=13&search=Frankie+Says+Relax&show=24 There's all sorts of searchresults.asp page being flagged. Is this something i can safely ignore or is it something i should endeavour to rectify? I'm also getting errors reported on shoppingcart.asp pages as well as pindex.asp (product index). I'm thinking i should maybe add disallow/ shoppingcart.asp to my robots text file, but am unsure as to whether i should be blocking robots from the search results pages and product index (which is essentially a secondary sitemap). Any advice would be greatly appreaciated. Thanks, Dave 🙂
Moz Pro | | giddygrafix0 -
Is there any report / tool that gives me last cache date for each page on my site ?
my site has several hundred pages, and it is important for me to know last crawl date of each page as well as number of pages cralwed in a particualr period ( from / to date ). is there any report in seomoz that can help for this ? or any other suggestion ?
Moz Pro | | elegantmicroweb0 -
Finding the source of duplicate content URL's
We have a website that displays a number of products. The product has variations (sizes) and unfortunately every size has its own URL (for now anyway). Needless to say, this causes duplicate content issues. (And of course, we are looking to change the URL's for our site as soon as possible) However, even though these duplicate URL's exist, you should not be able to land on them by navigating through the site. In theory, the site should always display the link to the smallest size. It seems that there is a flaw in our system somewhere, as these links are now found in our campaign here on SEOmoz. My question: is there any way to find the crawl path that lead to the URL's that shouldn't have been found, so we can locate the problem?
Moz Pro | | DocdataCommerce0 -
Duplicate Content Issue from using filters on a directory listing site
I have a directory listing site of harpists and have alot of issues coming up that say: Content that is identical (or nearly identical) to content on other pages of your site forces your pages to unnecessarily compete with each other for rankings. Because this is a directory listing site the content is quite generic.The main issue appears to be coming from the functionality of the page. It appears that the "spider" is picking up each different choice of filter as a new page? If you have a look at this link you will see what I mean. People searching the site can filter the results of the songs played by this harpist by changing the dropdowns etc... but for some reason the filter arguments are being picked up...? Do you have any good approaches to solving this issue? A similar issue comes from the video pages for each harpist. They are being flagged as identical content - as there are currently no videos on the page. | http://www.find-a-harpist.co.uk/user/39/videos | http://www.find-a-harpist.co.uk/user/37/videos | Do you have any suggestions? Many thanks for taking the time to read this and respond. | | | | | |
Moz Pro | | dseo241
| |0 -
What are followed vs non-followed links in Domain Analyis report?
I'm new to SEOMoz and wondering if some nice person can put this into plain English for me? 🙂 On the Competitive Domain Analysis report there are green follow / no-follow links and blue followed / non-followed linking root domains? what is the context of the green & blue circles? is there a better or worse visual representation that I should aim for - more dark green, more light green or about half & half. thanks in advance
Moz Pro | | seanuk0 -
Press Release - using moz bar/OSE is reading domain not page? How? Why?
A question posed by Christopher Glaeser from early today:low PA high DA, had a follow up response from him providing 2 urls from PR WEB for separate press releases: http://www.prweb.com/releases/2011/11/prweb8923419.htm (HP White) On moz bar Page Analysis/Link Data = PA - 47 DA - 36 http://www.prweb.com/releases/2011/12/prweb9051351.htm (Golfer's Advice) On moz bar Page Analysis/Link Data = PA - 1 DA - 96 I kept scratching my head as to how a press release of 6 weeks ago had garnered such attention from a company that would not seem to have a huge traffic due to more obscure product offering and scientific subject (Analyses of Armor Industry versus Golf Advice).
Moz Pro | | RobertFisher
Then I realized that for HP White, Link Data was not about the PR. The url from mozbar was HPWhite.com not the above, I did not notice until I used OSE where same thing was happening. When I cut and pasted the above press release url for HP White and placed it in OSE this changed: PA - 49 DA - 96 (2 links 2 linking root domains) For Golfers advice (0 links from 0 linking domains) Note to all: the links to the PR WEB release for the HP included a low end directory type link and a link from PR WEB (. For Golfer's Advice there was not a link back to the release from PR WEB: Note that Golfer's Advice is a newer release (6 weeks). So, any link from HPWhite release would equal more juice to HP White and PR Web and Vocus. Any link to Golfer's Advice from release offers......???? to Golfer's Advice and who cares to Vocus and PR Web. So, I guess this begs a couple of questions: Why the mozbar link analysis difference for one versus the other? Does PR Web treat some differently than others? Who benefits most from me paying a PR Web to do press releases for a client, PR Web and Vocus or my client and I???? I have tried to order the images to make sense: L to R top, then bottom is last. [](<a href=)" target="_blank">a> [](<a href=)" target="_blank">a> [](<a href=)" target="_blank">a> [](<a href=)" target="_blank">a> [](<a href=)" target="_blank">a> [](<a href=)" target="_blank">a>0