Moz Crawl shows over 100 times more pages than my site has?
-
The latest crawl stats are attached. My site has just over 300 pages?
Wondering what I have done wrong?
-
total pages is higher you are right Keri but still only 581
-
I believe this image looks at what's indexed that's a subset of your sitemap that you submitted. You may want to look at Google Index -> Index Status in GWT to see what it shows there.
-
latest Moz crawl
-
latest webmaster tools crawl
-
I will definetly be paying attention to those numbers Keri. Webmaster tools is showing the right number of pages (something over 300 with 90% of those indexed)
-
It's not going to be a penalty, but it'll be good to have a bit less of a load on your server (bots no longer crawling thousands of pages) and just have your real pages in the index.
Places to look for interesting changes in site metrics would be your organic traffic in analytics and taking a look at your Google Webmaster Tools account to see your impressions, pages crawled, etc.
-
Thanks Keri, I will update asap.
could you let me know how big an issue would this be? (When you have the time of course;))
-
You're welcome! I may have opened a can of worms, however. That sitemap is generated by an automated tool (based on the footer at the bottom), so somehow it's finding that page 28 as well.
You may also want to ask the developer if you should be indexing the categories in the blog archives. There are resources on Moz about the best way to set that up in Wordpress, but I don't have them at my fingertips at the moment (I have a snuggly baby sleeping on my lap instead that's slowing me down a tad).
To answer your next question, after you figure out where the page 28 is being linked from and cure that, yes, you can do a one-time crawl from Research Tools. It won't overwrite your campaign info, but you can at least see if Moz is seeing thousands of pages or just a few hundred to see if stuff was fixed. Again, happy to provide more detail if/when you need it (and others will likely jump in with help on the thread, too).
I'd love to also see a little update a few weeks down the line of any changes you've noticed on your site metrics after getting this fixed.
-
You rock:)
-
And I found it. The sitemap at http://www.nineclouds.ca/sitemap includes a page /28, which is where the crawlers are finding the non-existent pages.
-
If you look at http://www.nineclouds.ca/blog/page/23, you'll see that there's a double arrow in the pagination at the right that goes to page 24, even though the last page is page 21. Google somehow has found the pages greater than 21 (which I'm not sure how they found), and once they found one of those, they keep seeing the link there with the double arrows to go to another page. Same happened with Rogerbot. I'm not sure where the bad originating link is (what legit page on your site is linking to something over page 21), but that's the loop that's happening and causing a ton of pages to be indexed. Get rid of those, and you'll also get rid of most of your errors.
-
Not shy about that at all thanks Keri.
any help you can provide is greatly appreciated.
-
Hi Bill,
Using my admin powers, I took a peek at your account. I'm still trying to figure out where it's coming from, but you have thousands of empty pages of your blog indexed. I'll dig around a little more and see if I can figure out what's up.
If you're comfortable with sharing your URL here in a public forum, other people can come take a look too. Otherwise, I'm happy to send you a private message with part of what's up and give your developer a place to start looking.
-
Thanks Keri. I am the owner of the site not the programmer so I am looking up the terms you are using as I write this response. If I am using pagination is there a way for the moz not to allow for this? If I understand your question about the calendar correctly I do have one as part of my blog that dates each post? Can I get the bot to not recognize this calendar?
-
My first guess would be parameters or something are being crawled. Do you have pagination? Sorting ascending and descending? A calendar that's getting crawled through the year 2525?
Your next step would be to look into what those duplicate pages are and see if something is amiss that's generating a ton of URLs.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Tracking Clicks on a Global Header Across Multiple Sites
Hey All, A particular client has multiple websites and we're planning on implementing a global header across 15+ sites. I've been looking for a way to track the clicks on this global header across all sites (that is that they are summed up), what's the best way to go about this if I am using Google Analytics (I know Adobe site catalyst could do this no problem with some advanced tweaking), any ideas? I could do the general click tracking route and tag every link but that will only help me if I do that for each site (that being said, if the global header for all sites pulls from a single HTML, then tagging it would technically count all the clicks from all the sites, the only caveat being that I'd have to pick which Google analytics profile I'd want to track the header with). Thoughts? Thanks!
Reporting & Analytics | | EvansHunt0 -
Google analytics - landing pages
Hello, I have a website and it contains landing pages. after tracking and analyzing the data in Google Analytics , I found out that the traffic for the whole site is affected by the visits to the landing page. I mean, the bounce rate is less than one minute. this is because users fill their details in the landing pages and leave my website. to improve my website, I need clear data about the traffic, bounce rate etc. in my website. I also need can you help me solve this problem I thought about create another account in Google Analytics and set it to my websites pages (not including the landing pages). and keep the current code for the landing pages. my question is regarding this: Is it legal to use 2 different GA code (each one of them belong to different account) in the same domain ? can you provide me with more information about Multi Account in Google Analytics, and how can I use it to divide the traffic in my website between the traffic and the data for the landing pages and for the website itself ?
Reporting & Analytics | | JonsonSwartz0 -
Why is GG analytics showing so much (not provided)?
is there a better analytics tool (preferably free) that will show the actual keywords that drive traffic to a website or are we doomed to deal with (not provided) as a result of the Big G's retarded privacy policy? - seems they cant make up their minds as to whether they are going to take care of site owners or advertisers - but they should really take a stand and stop putting out half-ass results if anyone has any helpful ideas that may ease my pain i would be forever in your debt.
Reporting & Analytics | | Ezpro90 -
Filter out IP address of Site Search analytics
Hi Mozzers, I have a filter that excludes all internal traffic from my sites. But this does not seem to work on site search > Search Terms See here:- http://productforums.google.com/forum/#!searchin/analytics/filter$20site$20search/analytics/pO18L31hEO4/tJ3lKVNT3YYJ Any ideas? Or is it a bug, etc Thanks S
Reporting & Analytics | | Metropolis0 -
How Can a Page Have More Unique Pageviews than Total Pageviews?
The question and the attached image say it all... how is this possible? 2v1uo8w.jpg
Reporting & Analytics | | glennfriesen0 -
What is this referrer site?
Hi Guys....i keep seeing this in my analytics..can someone tell me what it is? 146w.bay146.mail.live.com thanks for your time
Reporting & Analytics | | nomad-2023230 -
Weekly crawl or monthly crawl?
Dear SEOmoz, Thanks for all the stuff in this pro account. I tweak my websites everyday and do a lot of linkbuilding. But I noticed the statistics don't change weekly. I've been using SEOmoz tools now for maybe two months and there has been one update in stats since. Am I too impatient or is it a monthly crawl? ^^ Thanks again! PS: Mostly talking about the opensiteexplorer.org statistics
Reporting & Analytics | | erikwigman0 -
Googlebot encountered extremely large numbers of links on your site??? How Do I resolve this?
I am working on a site with over 30 million pages. Every time I get about One Million indexed I get a Message in the Google Webmasters Tools saying "Googlebot encountered extremely large numbers of links on your site" The indexing then starts dropping like a Rock. I need to get the site indexed. Please Help!
Reporting & Analytics | | GlobalFlex0