Crawler Stats
-
Hello,
On all of my crawler stats, it is showing both http://domainhere.com and http://www.domainhere.com
Is it bad to have both types of URLS (one with www and one without www) or should we be only using links with www?
This question may not make any sense, but pretty much should it matter that both versions are showing up in my crawler results or not?
Thanks for any help in advance.
-
This article about canonicalization might help:
-
This article about canonicalization might help:
-
It make take a full crawl or two before it fixes itself. Keep me posted.
-
Thanks for the reply. I am going to change it to your method now and see what happens.
Thanks!
-
I'm not sure by looking at it if it's wrong since I'm not really an Apache coder. But I'm just guessing it's probably wrong since the bots are still detecting both versions of your site. I have it the way I showed you in my .htaccess file and I don't have any problems with bots detecting multiple versions. Maybe try it this way. You can always make a backup of your current file and switch back if it doesn't make a difference.
-
RewriteEngine on rewritecond %{http_host} ^domainhere.com [nc] rewriterule ^(.*)$ http://www.domainhere.com/$1 [r=301,nc]
This is what I currently have in my .htaccess file - is this wrong, and should I change it to the method that you posted?
Any information is appreciated. Thanks!
-
Yes, it matters. It could mean that Google is detecting it that way too, and may view them as two different sites with duplicate content. This could be hurting your rankings. Do you have access to your .htaccess file? You should add this code:
RewriteEngine On
RewriteCond %{HTTP_HOST} !^www.domainhere.com$
RewriteRule (.*) http://www.domainhere.com/$1 [R=301,L]This will redirect the non-www version to the www version and tell the crawlers this is how you'd prefer your site to display.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Critical crawler errors...4xx
Hey fam, I ran the Critical Crawler Issues and found 9 pages with critical crawler issues. I'm running a wordpress site and looked in the dashboard for Pages and Posts but the links aren't in the dashboard. Can you help fix? Thank you!
Technical SEO | | Myflgreen0 -
What crawler do you recommend for finding orphaned pages on a website?
Is there a crawler that you guys recommend for finding all pages, including orphaned pages on a website? A data export is not feasible. I saw a question from back in 2013 and was wondering if anything has changed since then in regards to crawling orphaned pages. Do most enterprise systems already have this built into their crawler? Or is it best to get a crawler like Xenu or Screaming Frog or Deepcrawl?
Technical SEO | | DigitalMarketingSEO0 -
Organic search traffic stats "leaking" into other channels?
Hi Everyone I have a website and am slowly getting to grips with SEO. Last week I enabled a new channel in google analytics which was "email" so I could track effectiveness of the weekly emails we send out. The good news is that a ton of traffic is now being assigned to the email "channel" in GA but my organic search traffic in channels is now down week on week. That feels odd as my overall traffic to the site is up, week on week. Does anyone have any experience of new channels coming on stream and canniballising old ones? Could it be that some of the traffic associated to organic search previously was actually coming from my email, I just didn't know it? thanks all!
Technical SEO | | NappyValleyNet1 -
Why are crawlers not picking up these pages?
Hi there, I've been asked to audit a new subdomain for a travel company. It's all a bit messy, so it's going to take some time to remedy. However, one thing I couldn't understand was the low number of pages appearing in certain crawlers. The subdomain has many pages. A homepage, category pages then product pages. Unfortunately, tools like Screaming Frog and xml-sitemaps.com are only picking up 19 pages and I can't figure out why. Google has so far indexed around 90 pages - this is by no means all of them, but that's probably because of the new domain and lack of sitemap etc. After looking at the crawl results, only the homepage and category (continent pages) are showing. So all the product pages are not. for example, tours.statravel.co.uk/trip/Amsterdam_Kings_Day_(Start_London_end_London)-COCCKDM11 is not appearing in the crawl results. After reviewing the source code, I can't see anything that would prevent this page being crawled. Am I missing something? At the moment, the crawl should be picking up around 400+ product pages, but it's not picking up any. Thanks
Technical SEO | | PeaSoupDigital0 -
Traffic stats disaster
Hi, We have recently moved a few old websites to an existing one and redesigned the existing website too. The visits in Analytics have fallen though the roof in the last few days. I would expect the traffic to have gone up because visitors from the old websites should now be on the redesigned one. Not sure what the problem is entirely. GA code is on the site but not before the closing head where I asked for it to be put. It's just before closing body tag. Does it make a difference? (It's a CMS site). GA code must be working because visits are being recorded (tens rather than the expected hundreds). Redirects have not been done properly - not sure what IT have done there. Would the lack of proper redirects be the sole reason for the drop? I am still hassling for 301's to be sorted. Thanks all
Technical SEO | | Houses0 -
SEOMoz Crawler vs Googlebot Question
I read somewhere that SEOMoz’s crawler marks a page in its Crawl Diagnostics as duplicate content if it doesn’t have more than 5% unique content.(I can’t find that statistic anywhere on SEOMoz to confirm though). We are an eCommerce site, so many of our pages share the same sidebar, header, and footer links. The pages flagged by SEOMoz as duplicates have these same links, but they have unique URLs and category names. Because they’re not actual duplicates of each other, canonical tags aren’t the answer. Also because inventory might automatically come back in stock, we can’t use 301 redirects on these “duplicate” pages. It seems like it’s the sidebar, header, and footer links that are what’s causing these pages to be flagged as duplicates. Does the SEOMoz crawler mimic the way Googlebot works? Also, is Googlebot smart enough not to count the sidebar and header/footer links when looking for duplicate content?
Technical SEO | | ElDude0 -
How does your crawler treat ajax links?
Hello! It looks like the seomoz crawler (and google) follows ajax links. Is this normal behavior? We have implemented the canonical element and that seems to resolve most of the duplicate content issues. Anything else we can do? Example: Krom
Technical SEO | | AJPro0 -
CSS Hiding Text - Does this matter to search engine crawlers
Hello, I'm working on a site and a developer is using CSS to mask crawlable links below. Then, java, advanced search links go on top of this. So, if you disable Java, but have CSS enabled, you don't have a lefthand nav. With both CSS and Java disabled you have a fully crawlable website. Is this a red flag? I understand a user without java would have a problem since most people don't disable CSS. But, is this a problem for search crawlers? Thanks!
Technical SEO | | Blenny0