Why does Moz only seem to be crawling a snap shot of the site I am working with?
-
I was wondering if anyone can help? I am working using Moz to help improve the SEO on a website I am working with, the website contains thousands of pages, yet for some reason Moz only seems to be crawling a small snap shot of the website.
I know there are particular pages that I had added a couple of weeks ago - about 300 in total - and none of these were showing on the first crawl, so I did another on-demand crawl and some of these showed up then. Despite this, it says it crawled 700ish pages, but there are getting close to 20-30ish thousand live pages on the site.
Any thoughts and guidance as to why they crawling may be stopping?
-
Hi there,
many thanks for your message. The site is an e-commerce website which currently has just short of 8000 products (each on their own page). These pages are all presented in categories to be able to click on, therefore I would imagine that this is all set up with internal links based on that.
There are numerous products showing, but only 700 pages are coming up in Moz. its weird, because I was expecting some duplication errors on a range that I added, yet these aren't showing in the crawl - but they are definitely there on the site, and under several categories that have links to the categories.
I'm a little stumped at why it doesn't seem to be crawling the full site.
-
Hi there,
Are you internally linking to all these pages that you want Moz to discover? It is possible that those pages are orphaned and they do not have any internal links pointing to them, so Moz cannot crawl them.
Ross
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Not Crawling Angular SPA
I have a client that just launched a redesigned website using Angular as a single page app. Google appears to be able to crawl the site just fine, but Moz crawl is only finding one page. We have updated the htaccess to allow for Rogerbot and Dotbot, but still unable to crawl any pages other than the home page. Does anyone have experience with this or ideas of why it won't crawl all pages, and how to allow for Moz to crawl all pages? There is a sitemap with approx. 390 pages. Thanks!
Getting Started | | PIN_Celler1 -
Does MOZ pick up every issue in one crawl?
Hi, Does MOZ pick up every error/warning in one crawl? Or does it take numerous crawls? Many thanks Lee
Getting Started | | lbagley0 -
What to believe Google's Comp or Moz's Opportunity/Potential graph
Hi! I'm newbie and I really don't know which info is best to dial in with the keywords. Google tells me that my keyword is in high competition, but Moz suggests that I have very high opportunity and potential with low difficulty with the same keyword... Maybe I need some sleep and come back later :-). Many thanks!
Getting Started | | PapaMilicia0 -
Multiple Google Analytics Accounts connected to the same MOZ subscription
Hi everyone, I have a quick question regarding setting up my MOZ account and it's
Getting Started | | Bernardo.Reguero
integration with a Google Analytics account: I'm not sure if I'm right but I think I can only set a single Google
Analytics account to do my analysis and then created several campaign. Or is there a possibility to set several Google Analytics accounts and creating several camapign to track diferente websites (connected to diferente Google Analytics accounts) ? Thanks! Cheers, Bernardo0 -
My website does not allow all crawler to crawl, Now my question is that whether i need to give permission to moz crawler if yes then whaat is moz bot name?
My website does not permit all crawler to crawl website. Whether ii need to give permission to moz bot to crawl website or not? If yes what is the moz bot name?
Getting Started | | irteam0 -
Why can't I Ctrl + click on links on Moz any more?
I'm interested if it's just me that gets frustrated by this? I've just Ctrl + clicked a few links to open them in separate tabs and then realised that none of them had opened. I know it's been like this for a while. It's a usability issue as it goes against expected norms, and now I have to right-click and then click "Open in new tab" on each link, which is more time-consuming and frustrating. More and more websites seem to be losing their Ctrl + click on links ability (JavaScript often breaks it). I don't know if there's a Mac equivalent... Anyway, I hope that doesn't seem like I'm too angry. It just frustrates me a little and I hope it gets fixed. 🙂 Edit - I've just realised these are getting blocked by Chrome's pop-up blocker - but why? It's only an issue on a small number of websites.
Getting Started | | Alex-Harford1 -
Moz Pro Trial...."Oops! Try refreshing the page"
I am in my MOZ PRO trial. I want to add a campaign to see how I like it, but something keeps coming up saying "Oops! Try refreshing the page, if that doesn't work click here to contact our help team" And I can't find an answer to that question here. Please help ASAP Ld3rHLi
Getting Started | | WFC-Sales0 -
Moz Analytics set up: should I use www or not when I do website set up?
Hello, When doing website set up will www.mysite.com yield different results than just mysite.com ? Same about competitors: should I use www or not when I set up their sites? Or it won't matter? Thanks,
Getting Started | | dstepans
Denis0