SeoMoz crawler giving false positives?
-
SeoMoz crawler indicated a few times that my site has a duplicate home page error (http://mysite.com and www.mysite.com)
I eliminated the the couple remaining internal links that pointed to http://mysite on a couple pages (all other internal links point to http://www.mysite.com)
I ran the crawl again and it said no errors this time. I naturally thought the duplicate page error problem was fixed.
However this morning I got the regularly scheduled crawl report from SeoMoz that said again I have those duplicate error pages. No changes were made to any of my site's pages between the crawls.
That makes me wonder if the crawler is providing false positives at times or was wrong when it said on the crawl a couple days ago that I don't have any errors (no duplicate page error).
Now, I don't know what to think.
-
Hey,
Our crawler actually requests the page http://mysite.com first but then finds all your links to www.mysite.com
You will want to contact the person responsible for hosting or developing your site in order to make these changes.
Have a great day!
Kenny
-
Thanks for the explanation. Could you answer a couple questions?
1 - If all internal site links go to www.mysite.com (none link to http://mysite.com), how does a duplicate page even happen? I don't understand how this happened to begin with if I don't have any such internal link to http://mysite.com.
2 - Can you recommend a service who can fix the htaccess page for me to create the 301 redirect? I'm not sure I want the hosting service doing it and making a mistake.
Thanks!
-
Hey,
That third campaign is actually a subdomain setup to crawl non-www. No duplicate content errors were presented because there are not any links to follow since all the links contain the subdomain www in them.
Root domain campaigns are distinguished with an astrick before the domain name.
-
Thanks - I initially thought that was it.
But if you see my 3rd campaign of the crawl, it runs it for the root domain and it shows no duplicates.
-
Hey,
I just looked into the issue that you are experiencing with our crawler. The reason the the discrepancy is because you actually have two separate campaigns running for the same site. One is set to crawl the root domain and one the subdomain.
The root domain campaign actually still presents these errors and has week over week but the sub-domain campaign is setup for the www version of your site and that's why these errors are not present, because the crawler won't even attempt to crawl off of www.
It is advisable to perform a 301 redirect as the other commenters mention.
Hope that helps!
Kenny
-
My point is the inconsistency in the SeoMoz crawler reports.
I got two SeoMoz crawl reports today - one was the regularly scheduled one which said I have duplicate home pages (as noted) and the crawl I started a couple hours ago said there are no errors.
So...how do you tell which one is right? Both cannot be since there were no changes to my website pages between the crawls.
thx
-
Hi,
If needed - this is the .htaccess code to help fix this issue; (Make sure and back up .htaccess before making any chages)
Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_HOST} ^yourdomainhere.com [NC]
RewriteRule ^(.*)$ http://www.yourdomainhere.com/$1 [L,R=301]
The above code would redirect all traffic from non www to www version of your site fixing dup content issues in that regard
Source ;http://www.webconfs.com/how-to-redirect-a-webpage.php
PS Spaces between lines not needed (funky formatting here)
Hope this helps
-
You need to redirect one of your home pages to the other. www.mysite.com is different to the crawl robot as my site.com. In addition to having the issue with seomoz, you are losing serp value for your home page because you are dividing up the SEO value. Do a 301 redirect from one to the other and voila....problem solved.
Please make sure you give me the thumbs up for the help!! Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz crawler is not able to crawl my website
Hi, i need help regarding Moz Can't Crawl Your Site i also share screenshot that Moz was unable to crawl your site on Mar 26, 2022. Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.
Technical SEO | | JasonTorney
my robts.txt also ok i checked it
Here is my website https://whiskcreative.com.au
just check it please as soon as possibe0 -
Product schema with no offer as owner wants to give price per customer
Hi, Trying to markup products for a site that does not show prices. Is there any way to markup a product price when the business model is: 1. customer calls or contacts shop. 2. shop gives a price quote based on level of detail and finish on the product 3. there is no base or top price. Thanks in advance!
Technical SEO | | plahpoy0 -
Basic SERP report does really gives me useful information?
Hi There! Ive seen so many times in the Basic SERP report, that pages/domains with low values are at the TOP of the SERP report, and i dont really understand that why can this be? I also have the same low values but i cant move higher for a keyword and i thought there is no question, i should improve my content/site, etc... But still dont understand why i see so many times that other domain/pages with lower values are at the top? On the attached screenshot, i am in the 12th position... Thx E. serp_report.jpg
Technical SEO | | Neckermann0 -
Blogspot domains - giving me a manual action
So some agency did horrendous article submissions on mass in 08/09. Since I have been tidying this up by manually getting the domains removed in our back-link profile. Some however i just cannot get rid of. Recent penguin update obviously penalised me for this, so i disavowed the rest i could not remove and did a reconsideration request. The reply from Google was still that it violates guidelines and it used 3 blogspot domains (which no crawler i used had previously found) as examples. Now there is NOONE at Google to contact about this and the sites are abandoned, so they just sit there doing damage. I will ofcourse add these to the disavow but can i disavow the whole of blogspot.com ? What if all are in the disavow but they still use it against us in the reconsideration request and i cannot remove them as noone to contact at Google? Really appreciate the help, thanks, 2 years of hell tidying up bad agency work!
Technical SEO | | pauledwards0 -
My homepage+key pages have dropped 40+ positions after implementing redirects and canonical changes. HELP!
Hi SEOMozers, I work for a web based nonprofit at www.tisbest.org. I had a professional contact recommend that we work on our redirects to our homepage because we were losing valuable rank benefit. This combined with getting sick of seeing our weekly SEOMoz crawl reports show 304 duplicate page and title errors for months. No one could seem to figure out what was happening (we think it had to do with session stuff; we were seeing several versions of each page showing the following: www.tisbest.org/default.aspx/(random character string) My developer and I read a bunch of articles and started making changes 10 days ago: He setup 301 redirects from http://tisbest.org to http://www.tisbest.org. (set the canonical domain). We did a redirect from http://www.tisbest.org/default.aspx to root with "/". I set the canonical setting to www.tisbest.org in our webmaster tools. In our web config (we're running in asp.net), we changed our session detection from auto-detect then saw some session funkiness so we changed it back. Though we do think the character strings we were seeing were session GUID. He forced lower case URL’s to reduce duplicate page content/titles. I got my weekly crawl report 9 days ago and we had dropped from 340 duplicate page title and page content errors went to one. We went nuts and felt like the kings of SEO. Then, yesterday (9/28), the SEO grim reaper came knocking when I received my weekly SEOMoz ranking report. It said we had dropped 40+ spots for all of 9 of our keywords. Sure enough, I searched our keywords and our website was gone. Then I searched our company name, tisbest, and only a few of our pages show but not the homepage. I searched for our URL www.tisbest.org, and I originally got the expanded view (with 8 links to various webpages - can't remember what this view is called) but now, today (Saturday), the expanded view is gone from this search result. Also, when I run the On Page Report card for our homepage, I get the following error message with no results: "We were unable to grade that page. The page did not load. Curl::Err::TooManyRedirectsError: Number of redirects hit maximum amount." When I run the Open Site explorer report, I get this message at the top: Oh Hey! It looks like that URL redirects to www.tisbest.org/?AspxAutoDetectCookieSupport=1. Would you like to see data for <a class="clickable redirects">that URL instead</a>?" If I go to the report for the that report's page, it says that "No information is available for that URL." Just tonight (night of 9/29), our developer added the rel="canonical" href="http://www.tisbest.org" /> to our homepage tonight to see if that would help. We did not do that originally. In our Google Webmaster tools, I am seeing the number of URL Error - Not Followed has sky rocked. I have attached a screen capture to this thread. There are also a large number of URL Errors - Not Found errors as well. I did some research tonight and downloaded and ran Screaming Frog SEO Crawler. I have attached a screen capture below with this report and a couple of questions I sent our developer that may be helpful to you. Also, not sure if this is relevant, we use a master page that all of our pages inherit from so all of our pages get the same meta-data: name="keywords" content="charitable gift card, charitable gift certificate, non profit gift card, charity donation, giftcard, charity gift card, donation gift card, donation gift, charity gift, animal gift card, animal gift, environmental gift card, environmental gift, humanitarian gift card, humanitarian gift, christian gift card, christian gift, catholic gift card, catholic gift, religious gift card, religious gift" />id="ctl00_metaDescription" name="description" content="Award winning Charity Gift Card, for over 250 premier charities. A customized donation gift that makes the world better. TisBest is BBB Accredited." />name="google-site-verification" content="EfJIhN3h2SVSXdSpUbfceBVw2q6zrGX8rRQhdNZ1xY8" /><title></span><span> </span></p> <p>Can anyone help me/us identify the issue that obliterated our rankings? I am happy to give an information needed. Thank you! Chad Edwards</p> <a download="Bqcu1.png" class="imported-anchor-tag" href="http://i.imgur.com/Bqcu1.png" target="_blank">Bqcu1.png</a> <a download="ZXQ8d.png" class="imported-anchor-tag" href="http://i.imgur.com/ZXQ8d.png" target="_blank">ZXQ8d.png</a></title>
Technical SEO | | TisBest0 -
Why would SEOMoz and GWT report 404 errors for pages that are not 404ing?
Recently, I've noticed that nearly all of the 404 errors (not soft 404) reported in GWT actually resolve to a legitimate page. This was weird, but I thought it might just be old info, so I would go through the process of checking and "mark as fixed" as necessary. However, I noticed that SEOMoz is picking up on these 404 errors in the diagnostics of the site as well, and now I'm concerned with what the problem could be. Anyone have any insight into this? Rich
Technical SEO | | secretstache0 -
Top pages give " page not found"
A lot of my top pages point to images in a gallery on my site. When I click on the url under the name of the jpg file I get an error page not found. For instance this link: http://www.fastingfotografie.nl/architectuur-landschap/single-gallery/10162327 Is this a problem? Thanks. Thomas. JkLej.png
Technical SEO | | thomasfasting0 -
Page not Accesible for crawler in on-page report
Hi All, We started using SEOMoz this week and ran into an issue regarding the crawler access in the on-page report module. The attached screen shot shows that the HTTP status is 200 but SEOMoz still says that the page is not accessible for crawlers. What could this be? Page in question
Technical SEO | | TiasNimbas
http://www.tiasnimbas.edu/Executive_MBA/pgeId=307 Regards, Coen SEOMoz.png0