Way has the number of pages crawled plummeted?
-
Why has the number of pages crawled for our campaign plummeted in Moz Analytics – down to 729 from over 10k? Don't see any issues in Google Analytics with crawling our site.
-
Wow, that was like an hour of digging--trying to track down the linkscape crawler. I think I've got it now.
-
Rogerbot is the campaign crawler, and OSE has a different name. I know it can be a tad bit confusing at times.
-
Ah huh, I see. So does the campaign crawler have a different name or does it go by Rogerbot too?
-
Chris, what you're talking about is Open Site Explorer. I believe what Eyeglass Guy is talking about is the campaign crawl, which is an entirely separate crawl system.
-
Eyeglasses,
I wonder if your Mozrank or Moztrust has ratcheted down, as well, or if you've made any architectural/navigational changes to the site. With that many pages, it is possible that its not getting crawled as deeply as it had in the past if those dropped pages no longer met the value criteria for crawling by rogerbot--maybe you lost a good link or two or something like that. Also, if you changed any internal linking, it might be that those pages may have been pushed down in the architecture and not be as accessible as they once were.
-
How many pages does your site have? Did you possibly clear up a bunch of parameters and duplicate content, so the crawl is more accurate?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Need help fixing a duplicate content issue for my website. The moz crawl is show OMG my website with https:// and https://www. But I have never used the url https:// so I don’t understand why moz is showing this
Moz is showing my url with two different starts. Https:// and then the one I use https://www. The problem is I don’t think I have ever used the url without the www. at the start. How do I fix this?
Moz Bar | | jdp_uk0 -
Moz Crawler Causing Server Timeouts... Crawling thousands of non-existant pages with query parameters
Moz crawler is crawling all pages like this: http://www.xxxx.com/?product_count=100&product_order=desc&product_orderby=date http://www.xxxx.com/?product_count=100&product_order=desc&paged=1 http://www.xxx.com/?product_count=100&product_order=desc&product_view=grid Last month it crawled 80,000 pages on a site with less than 100 pages. Is there a way to select only certain pages to be crawled? Right now it is still crawling this site, since Monday morning and it's Tuesday mid-day. Every Monday it is causing time-outs from high band width on our server. Just getting ready to delete this client from the account unless there is a solution someone can give us. Thanks.
Moz Bar | | adirondack0 -
On-page Grader API?
Hi guys, I have a massive list of keywords i want to optimise against a specific URL. I'm checking them manually with the MOZ on-page grader to determine if the keyword is optimised for the URL (Grade A score). I was wondering if there is a way to pull the scores automatically for each URL with the MOZ api? Or does anyone know of any tools which can automate this process? Cheers, Chris
Moz Bar | | jayoliverwright1 -
On Page Grader inconsistent
Why does the on page grader not update it's grades based on the other factors, other then the title tag. i.e. I can have the keyword 'burger' in H1 tags, the URL, ALT attributes, in the body text a couple times and in the meta section of the site and receive an F grade from the on page grader, but as soon as I add the word 'burger' to the title I receive an A grade. Is there a reason why the only factor that has that influence is the title tag, and why is it that a keyword for a page cant get say a B or C grade if it has the other factors covered (i.e. they have a tick next to them) but not the title tag? Cheers Again
Moz Bar | | sharpleaddesign0 -
Duplicate Home Page in Moz Report
I recently ran a crawl report in MOZ for www.laddersfree.co.uk and it has picked duplicate pages for my home page. The two pages are www.laddersfree.co.uk and www.laddersfree.co.uk The CMS System my developer is using is Concrete5. If you look at the source code there is a Canonical <link rel="<a class="attribute-value">canonical</a>" href="http://www.laddersfree.co.uk/" /> I am very confused as to why there are two pages picked up in the crawl. Can someone offer any advice please? Thank you. Jason
Moz Bar | | gymmad0 -
Moz Local | Empty page "Categories"
Dear Moz, Another error, the following url loads an empty page https://moz.com/local/categories Please review Thanks!
Moz Bar | | Bio-RadAbs0 -
Duplicate content reported for totally different pages
Hi, The Moz report is showing just over 21,500 duplicate page issues on our site. This is more or less every page we have. However when I look at the pages it says are duplicates they are totally different (it could for example report that a news page for 2009 is the same as a product page just added which has no relation when you read the content or view the page). What sort of thing could it be picking up as duplicate content? I assume it must be something in the HTML for the site rather than the actual page content as there is no cross over at all on the pages highlighted. The only issue I can currently identify is that the menu for the mobile version of the site has a huge number of internal links which I will cut down. If the tools purely look at HTML content this could be seen as duplicate but shouldn't it be clever enough to realise what is content and what is site structure? Thanks,
Moz Bar | | TW-Steve0 -
Not getting foreign characters in crawl diagnostics .csv
The crawl diagnostics .csv file is showing high-ascii characters instead of the correct language (foreign language website) e.g. Vietnamese, Chinese (both kinds), etc. Is there a way to get this right?
Moz Bar | | trainSEM0