Crawler Stats
-
Hello,
On all of my crawler stats, it is showing both http://domainhere.com and http://www.domainhere.com
Is it bad to have both types of URLS (one with www and one without www) or should we be only using links with www?
This question may not make any sense, but pretty much should it matter that both versions are showing up in my crawler results or not?
Thanks for any help in advance.
-
This article about canonicalization might help:
-
This article about canonicalization might help:
-
It make take a full crawl or two before it fixes itself. Keep me posted.
-
Thanks for the reply. I am going to change it to your method now and see what happens.
Thanks!
-
I'm not sure by looking at it if it's wrong since I'm not really an Apache coder. But I'm just guessing it's probably wrong since the bots are still detecting both versions of your site. I have it the way I showed you in my .htaccess file and I don't have any problems with bots detecting multiple versions. Maybe try it this way. You can always make a backup of your current file and switch back if it doesn't make a difference.
-
RewriteEngine on rewritecond %{http_host} ^domainhere.com [nc] rewriterule ^(.*)$ http://www.domainhere.com/$1 [r=301,nc]
This is what I currently have in my .htaccess file - is this wrong, and should I change it to the method that you posted?
Any information is appreciated. Thanks!
-
Yes, it matters. It could mean that Google is detecting it that way too, and may view them as two different sites with duplicate content. This could be hurting your rankings. Do you have access to your .htaccess file? You should add this code:
RewriteEngine On
RewriteCond %{HTTP_HOST} !^www.domainhere.com$
RewriteRule (.*) http://www.domainhere.com/$1 [R=301,L]This will redirect the non-www version to the www version and tell the crawlers this is how you'd prefer your site to display.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz crawler is not able to crawl my website
Hi, i need help regarding Moz Can't Crawl Your Site i also share screenshot that Moz was unable to crawl your site on Mar 26, 2022. Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.
Technical SEO | | JasonTorney
my robts.txt also ok i checked it
Here is my website https://whiskcreative.com.au
just check it please as soon as possibe0 -
Critical crawler errors...4xx
Hey fam, I ran the Critical Crawler Issues and found 9 pages with critical crawler issues. I'm running a wordpress site and looked in the dashboard for Pages and Posts but the links aren't in the dashboard. Can you help fix? Thank you!
Technical SEO | | Myflgreen0 -
Organic search traffic stats "leaking" into other channels?
Hi Everyone I have a website and am slowly getting to grips with SEO. Last week I enabled a new channel in google analytics which was "email" so I could track effectiveness of the weekly emails we send out. The good news is that a ton of traffic is now being assigned to the email "channel" in GA but my organic search traffic in channels is now down week on week. That feels odd as my overall traffic to the site is up, week on week. Does anyone have any experience of new channels coming on stream and canniballising old ones? Could it be that some of the traffic associated to organic search previously was actually coming from my email, I just didn't know it? thanks all!
Technical SEO | | NappyValleyNet1 -
Traffic stats disaster
Hi, We have recently moved a few old websites to an existing one and redesigned the existing website too. The visits in Analytics have fallen though the roof in the last few days. I would expect the traffic to have gone up because visitors from the old websites should now be on the redesigned one. Not sure what the problem is entirely. GA code is on the site but not before the closing head where I asked for it to be put. It's just before closing body tag. Does it make a difference? (It's a CMS site). GA code must be working because visits are being recorded (tens rather than the expected hundreds). Redirects have not been done properly - not sure what IT have done there. Would the lack of proper redirects be the sole reason for the drop? I am still hassling for 301's to be sorted. Thanks all
Technical SEO | | Houses0 -
Webmaster tools crawl stats
Hi I have a clients site that was having aprox 30 - 50 pages crawled regularly since site launch up until end of Jan. On the 21st Jan the crawled pages dropped significantly from this average to about 11 - 20 pages per day. This also coincided with a massive rankings drop on the 22nd which i thought was something to do with panda although it later turned out the hosts had changed the DNS and exactly a week after fixing it the rankings returned so i think that was the cause not panda. However i note that the crawl rate still hasn't returned to what it was/previous average and is still following the new average of 10-20 pages per day rather than the 30-50 pages per day. Does anyone have any ideas why this is ? I have since added a site map but hasnt increased crawl rate since A bit of further info if it helps in any way is that In the indexed status section says 48 pages ever crawled with 37 pages indexed. There are 48 pages on the site. The site map section says 37 submitted with 35 indexed. I would have thought that since dynamic site map would submit all urls Any clarity re the above much appreciated ? Cheers Dan
Technical SEO | | Dan-Lawrence0 -
Google Webmaster Tool - Crawl Stats Query ?
Dear All, I have been looking at GWT Crawl Stats and wondering how should I be interrupting the crawl stats chart. AllI I see is 3 charts telling me a high , low and average for the below but I am wondering is there anything I really need to be looking for ?. Pages crawled per day Kilobytes downloaded per day Time spent downloading a page (in milliseconds) thanks Sarah
Technical SEO | | SarahCollins0 -
SeoMoz crawler giving false positives?
SeoMoz crawler indicated a few times that my site has a duplicate home page error (http://mysite.com and www.mysite.com) I eliminated the the couple remaining internal links that pointed to http://mysite on a couple pages (all other internal links point to http://www.mysite.com) I ran the crawl again and it said no errors this time. I naturally thought the duplicate page error problem was fixed. However this morning I got the regularly scheduled crawl report from SeoMoz that said again I have those duplicate error pages. No changes were made to any of my site's pages between the crawls. That makes me wonder if the crawler is providing false positives at times or was wrong when it said on the crawl a couple days ago that I don't have any errors (no duplicate page error). Now, I don't know what to think.
Technical SEO | | finalfrontier0 -
Page not Accesible for crawler in on-page report
Hi All, We started using SEOMoz this week and ran into an issue regarding the crawler access in the on-page report module. The attached screen shot shows that the HTTP status is 200 but SEOMoz still says that the page is not accessible for crawlers. What could this be? Page in question
Technical SEO | | TiasNimbas
http://www.tiasnimbas.edu/Executive_MBA/pgeId=307 Regards, Coen SEOMoz.png0