Way has the number of pages crawled plummeted?
-
Why has the number of pages crawled for our campaign plummeted in Moz Analytics – down to 729 from over 10k? Don't see any issues in Google Analytics with crawling our site.
-
Wow, that was like an hour of digging--trying to track down the linkscape crawler. I think I've got it now.
-
Rogerbot is the campaign crawler, and OSE has a different name. I know it can be a tad bit confusing at times.
-
Ah huh, I see. So does the campaign crawler have a different name or does it go by Rogerbot too?
-
Chris, what you're talking about is Open Site Explorer. I believe what Eyeglass Guy is talking about is the campaign crawl, which is an entirely separate crawl system.
-
Eyeglasses,
I wonder if your Mozrank or Moztrust has ratcheted down, as well, or if you've made any architectural/navigational changes to the site. With that many pages, it is possible that its not getting crawled as deeply as it had in the past if those dropped pages no longer met the value criteria for crawling by rogerbot--maybe you lost a good link or two or something like that. Also, if you changed any internal linking, it might be that those pages may have been pushed down in the architecture and not be as accessible as they once were.
-
How many pages does your site have? Did you possibly clear up a bunch of parameters and duplicate content, so the crawl is more accurate?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I see amount of traffic to a particular page on a website?
I am trying to figure out how much traffic goes to a particular page on a website. How would I go about finding this?
Moz Bar | | jbcorcoran6 -
Crawl Test is now On-Demand Crawl!
If you've been with Moz a while, you may have used our old Crawl Test tool. A year ago we launched an all new, campaign-based Site Crawl (with an entirely rebuild crawl engine), but Crawl Test fell into disrepair and we haven't had a solid tool for crawling non-campaign domains. I'm happy to announce that we've just launched an all new On-Demand Crawl, built on the new Site Crawl engine, with a UI that's focused on quick insights. Moz Pro Standard tier customers can run up to 5 crawls per month at 3,000 page per crawl (crawls are saved for 90 days), with per-month limits increasing at higher levels. Most On-Demand Crawls should run in a few minutes, making the tool perfect to get quick insights for sales meetings, vetting prospects, or analyzing competitors. We've written up a sample case study or logged-in customers can go directly to On-Demand Crawl. Try it out -- we'd love to hear your use cases (either here or in the blog post comments).
Moz Bar | | Dr-Pete6 -
Why is Moz Crawling More Pages Than My Site Actually Has?
Hi I have a site that only has 5k pages but Moz has crawled 50K pages on the site when I initiated the site crawl. I don't exactly know why Moz is reporting me back so many pages but I was wondering why this is and if any of you out in the Moz community know anything about this. Thanks
Moz Bar | | drewstorys0 -
Is there any way to automatically filter negative keywords out of Keyword Explorer results?
I use Keyword Explorer a LOT for fleshing out new campaigns. However, the results I get frequently contain queries with words that are in my negative keyword lists. Today, I solve this with export and some Excel wizardry but keep thinking how convenient it would be if there was a way to provide a list upfront of negative keywords and get pre-scrubbed results. Does this exist? If not, how can I escalate this as a feature request? 🙂
Moz Bar | | ngowdy1 -
My page cant be crawled by MOZ
Hi everyone, I have a page named zita.vn. Im using Moz to do SEO for the page, but when I use On-page grader in MOZ, I got an error that the page cant be accessed. Additionally, I got an issue with page crawling. Google search still crawl my page normally. Please help. Thanks a lot.
Moz Bar | | zita.vn0 -
Too Many On-Page Links Notice
When calculating the number of links on a page, are navigation links included in the total? I have all of my navigation links within the <nav>element. I would think that there are a lot of sites out there that easily exceed the 100 link recommendation if you add up nav and footer links. </nav>
Moz Bar | | Brando160 -
On the on page optimization page, I found out that there are 2 contributing factors which are opposite to each other. "No More Than One H1 Tag" and "Appropriate Keyword Usage in H1 Tag"
"No More Than One H1 Tag" and "Appropriate Keyword Usage in H1 Tag" If you fulfill one condition, the other one is not completed. If you consider Article heading as H1 then Moz do not detect keyword in the heading.
Moz Bar | | MoeezLodhi0 -
Moz "Crawl Diagnostics" doesn't respect robots.txt
Hello, I've just had a new website crawled by the Moz bot. It's come back with thousands of errors saying things like: Duplicate content Overly dynamic URLs Duplicate Page Titles The duplicate content & URLs it's found are all blocked in the robots.txt so why am I seeing these errors?
Moz Bar | | Vitalized
Here's an example of some of the robots.txt that blocks things like dynamic URLs and directories (which Moz bot ignored): Disallow: /?mode=
Disallow: /?limit=
Disallow: /?dir=
Disallow: /?p=*&
Disallow: /?SID=
Disallow: /reviews/
Disallow: /home/ Many thanks for any info on this issue.0