Testing for duplicate content and title tags

[SEOvet]

Hi there,

I have been getting both Duplicate Page content and Duplicate Title content warnings on my crawl diagnostics report for one of my campaigns. I did my research, and implemented the preferred domain setting in Webmaster Tools. This did not resolve the crawl diagnostics warnings, and upon further research I discovered the preferred domain would only be noted by Google and not other bots like Roger. My only issue was that when I ran an SEOmoz crawl test on the same domain, I saw none of the duplicate content or title warnings yet they still appear on my crawl diagnostics report. I have now implemented a fix in my .htaccess file to 301 redirect to the www. domain. I want to check if it's worked, but since the crawl test did not show the issue last time I don't think I can rely on that. Can you help please?

Thanks,

Claire

[SEOvet]

Thanks Joseph. Very helpful.

[jlane9]

Hello Claire,

I really don't think you are going to get those errors show up on the next crawl.

Just on another note after seeing your URL...

I would also code all of the links in the site with full absolute links.

I see the links as

<li> href=" /our-services/diabetes-support/get-started/"> Diabetes Supporta>li>

I would add the  **http://www.**  in front of all those links.

[SEOvet]

Thanks Jared, that's awesome.

[JaredMumford]

You can always check by testing in your browser but the best way is to check the header response to make sure the server is sending the proper response (a 301) - your landing pages look good (see below). I use Live HTTP Headers which is a firefox plugin - hers what it tell you:

http://pharmacy777.com.au/our-pharmacies/applecross-village/

GET /our-pharmacies/applecross-village/ HTTP/1.1
Host: pharmacy777.com.au
User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:15.0) Gecko/20100101 Firefox/15.0.1
HTTP/1.1 301 Moved Permanently
Date: Thu, 04 Oct 2012 03:23:17 GMT
Server: Apache/2.2.22 (Ubuntu)
Location: http://www.pharmacy777.com.au/our-pharmacies/applecross-village/

So the redirect is working. The only thing i noticed was that the home page instantly switched to www and didnt even return a 301 so it appears you may have implemented a redirect there outside of htaccess.

If your report is still showing duplicates make sure that its not the trailing slash. Your URLs can be loaded as such:

http://www.pharmacy777.com.au/our-pharmacies/applecross/

http://www.pharmacy777.com.au/our-pharmacies/applecross

The best way to find out if the SEOMoz report is counting these as dupes is to Export the crawl report to CSV (top right of crawl report). Then go all the way to the far right column called 'duplicate pages' and sort it alphabetically. This column will show you all of the duplicate urls for each particular URL row. Lots of times you can find little 'surprises' here - that csv report is priceless!

[SEOvet]

Hi Joseph,

Yes, I have done this test and it appears to be working. I just want to be sure I'm not going to be faced with a load of warnings when my next crawl runs on the weekend, as when I implemented the Webmaster preferred domain what happened was"

• implemented preferred domain

• rank crawl test - looked to have resolved the issue but then

• 4 days later the scheduled crawl report ran, all errors still present.

Luckily I told the client I had to wait for the report results and didn't tell think it was resolved after the crawl test looked OK!! This time I've run the crawl test and done the manual test you suggested, but I want to be able to feed back to the client today if I can (confidently), and I no longer trust the test.

Thanks very much for your answer, it's always good to have someone validate your own approach.

Cheers,

Claire

[SEOvet]

Hi Jared;

www.pharmacy777.com.au

Small oversight!!

[jlane9]

If I understand correctly you want to see if your re-direct has fixed your duplicate content issue.

Right now you still get the error ...

I would simply type the url in the address bar, with and without the www. and see if both show up.

If the re - direct is working then only one should show up , the other should re - direct immediately.

If they both show up do then your .htaccess code may have a mistake.

hope that helps.

[JaredMumford]

Update: Reporting can be historic - so you are probably looking at a report from an older crawl.

[JaredMumford]

Hi Claire - we need the url of your site to check the headers on the 301 redirect!

Definitely a good way to fix this is via htaccess like you are suggesting you did. When I get a new client its in the campaign startup list and it works well. Make sure there arent any other issues like the infamous trailing slash causing duplication. If you provide the URL a quick check can be made.