SeoMoz crawler giving false positives?
-
SeoMoz crawler indicated a few times that my site has a duplicate home page error (http://mysite.com and www.mysite.com)
I eliminated the the couple remaining internal links that pointed to http://mysite on a couple pages (all other internal links point to http://www.mysite.com)
I ran the crawl again and it said no errors this time. I naturally thought the duplicate page error problem was fixed.
However this morning I got the regularly scheduled crawl report from SeoMoz that said again I have those duplicate error pages. No changes were made to any of my site's pages between the crawls.
That makes me wonder if the crawler is providing false positives at times or was wrong when it said on the crawl a couple days ago that I don't have any errors (no duplicate page error).
Now, I don't know what to think.
-
Hey,
Our crawler actually requests the page http://mysite.com first but then finds all your links to www.mysite.com
You will want to contact the person responsible for hosting or developing your site in order to make these changes.
Have a great day!
Kenny
-
Thanks for the explanation. Could you answer a couple questions?
1 - If all internal site links go to www.mysite.com (none link to http://mysite.com), how does a duplicate page even happen? I don't understand how this happened to begin with if I don't have any such internal link to http://mysite.com.
2 - Can you recommend a service who can fix the htaccess page for me to create the 301 redirect? I'm not sure I want the hosting service doing it and making a mistake.
Thanks!
-
Hey,
That third campaign is actually a subdomain setup to crawl non-www. No duplicate content errors were presented because there are not any links to follow since all the links contain the subdomain www in them.
Root domain campaigns are distinguished with an astrick before the domain name.
-
Thanks - I initially thought that was it.
But if you see my 3rd campaign of the crawl, it runs it for the root domain and it shows no duplicates.
-
Hey,
I just looked into the issue that you are experiencing with our crawler. The reason the the discrepancy is because you actually have two separate campaigns running for the same site. One is set to crawl the root domain and one the subdomain.
The root domain campaign actually still presents these errors and has week over week but the sub-domain campaign is setup for the www version of your site and that's why these errors are not present, because the crawler won't even attempt to crawl off of www.
It is advisable to perform a 301 redirect as the other commenters mention.
Hope that helps!
Kenny
-
My point is the inconsistency in the SeoMoz crawler reports.
I got two SeoMoz crawl reports today - one was the regularly scheduled one which said I have duplicate home pages (as noted) and the crawl I started a couple hours ago said there are no errors.
So...how do you tell which one is right? Both cannot be since there were no changes to my website pages between the crawls.
thx
-
Hi,
If needed - this is the .htaccess code to help fix this issue; (Make sure and back up .htaccess before making any chages)
Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_HOST} ^yourdomainhere.com [NC]
RewriteRule ^(.*)$ http://www.yourdomainhere.com/$1 [L,R=301]
The above code would redirect all traffic from non www to www version of your site fixing dup content issues in that regard
Source ;http://www.webconfs.com/how-to-redirect-a-webpage.php
PS Spaces between lines not needed (funky formatting here)
Hope this helps
-
You need to redirect one of your home pages to the other. www.mysite.com is different to the crawl robot as my site.com. In addition to having the issue with seomoz, you are losing serp value for your home page because you are dividing up the SEO value. Do a 301 redirect from one to the other and voila....problem solved.
Please make sure you give me the thumbs up for the help!! Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Suddenly Many 301 Redirects captured by SEOMOZ
On April the 7th SeoMOZ captured 6000 301 redirect on my site, but I cant seem to understand how SEOMOZ finds these links Example http://www.iphonegadget.dk/dk/apple-tilbeh-r-36/ipad-tilbeh-r-219/bilholder-239/index-2-4a.html Makes a 301 Redirect to the following page beneath SEOMOZ says http://www.iphonegadget.dk/dk/apple-tilbeh-r-36/ipad-tilbeh-r-219/bilholder-239/index-2.html The weird thing is that both urls work, but if i browse my site in a normal matter this link will never be created i that way. The -4a in the end os the link is not the normal link structure on the site and has never been like that before. So how does SEOMOZ Create that link? http://www.iphonegadget.dk/dk/apple-tilbeh-r-36/ipad-tilbeh-r-219/bilholder-239/index-2-4a.html Also google only has the right one that are this one beneath http://www.iphonegadget.dk/dk/apple-tilbeh-r-36/ipad-tilbeh-r-219/bilholder-239/index-2.html People would normal come to the category with this url http://www.iphonegadget.dk/dk/apple-tilbeh-r-36/ipad-tilbeh-r-219/bilholder-239/ And page 2 would be http://www.iphonegadget.dk/dk/apple-tilbeh-r-36/ipad-tilbeh-r-219/bilholder-239/index-2.html AND NOT http://www.iphonegadget.dk/dk/apple-tilbeh-r-36/ipad-tilbeh-r-219/bilholder-239/index-2-4a.html Can anyone find out what is going on?
Technical SEO | | noerdar0 -
Will what happen to raven happen to SEOMoz?
With this new update coming out with raven marketing with this SEO Moz have to do the same update? Are is this something that Raven does and SEOMoz doesnt do with getting 3 rd party info?
Technical SEO | | MarketMotiveStudents0 -
Dear Support, my client is a large bank and his site is https. and seomoz does not give any data. what can i do ? thank you
Dear Support, what to do in case of https pages. seomoz does not give any data about it. please help asap. Thank you.
Technical SEO | | SebestynMrton0 -
Crawler Stats
Hello, On all of my crawler stats, it is showing both http://domainhere.com and http://www.domainhere.com Is it bad to have both types of URLS (one with www and one without www) or should we be only using links with www? This question may not make any sense, but pretty much should it matter that both versions are showing up in my crawler results or not? Thanks for any help in advance.
Technical SEO | | EQ-Richie0 -
Impact of "restricted by robots" crawler error in WT
I have been wondering about this for a while now with regards to several of my sites. I am getting a list of pages that I have blocked in the robots.txt file. If I restrict Google from crawling them, then how can they consider their existence an error? In one case, I have even removed the urls from the index. And do you have any idea of the negative impact associated with these errors. And how do you suggest I remedy the situation. Thanks for the help
Technical SEO | | phogan0 -
How to create GA Profile for SEOMoz
Hi ! I get this : Oh No! We did not find any Google Analytics profiles associated with this account. So I was wondering how to create GA Profiles for SEOMOZ. I'm admin of 3 websites in my GA accounts ... Thanks !
Technical SEO | | ovivo0 -
SEOMoz is indicating I have 40 pages with duplicate content, yet it doesn't list the URL's of the pages???
When I look at the Errors and Warnings on my Campaign Overview, I have a lot of "duplicate content" errors. When I view the errors/warnings SEOMoz indicates the number of pages with duplicate content, yet when I go to view them the subsequent page says no pages were found... Any ideas are greatly welcomed! Thanks Marty K.
Technical SEO | | MartinKlausmeier0 -
Ecommerce site with currency selectors giving dupe content?
Hi everyone,
Technical SEO | | BeachDude
One of my ecommerce sites uses BigCommerce. They have a feature where you can add different currency buttons to change the currency that the customer can shop as. This is great because if people from the UK visit our site, they can change the currency to their own rather than US. It just ads a variable on the end of the URL string to change the currency. However, in my webmaster tools I noticed that I think i am getting a bunch of duplicate content. For example, it thinks i have duplicate title tags for the following: domainname/pages/my-cool-widget.html
domainname/pages/my-cool-widget.html?setCurrencyId=1
.domainname/pages/my-cool-widget.html?setCurrencyId=2
domainname/pages/my-cool-widget.html?setCurrencyId=3
domainname/pages/my-cool-widget.html?setCurrencyId=4 I thought about adding "rel=no-follow" but unfortunately I don't have access to this file to edit the code. Any suggestions?0