Campaign Crawl
-
I have a site with 8036 pages in my sitemap index. But the MozBot only Crawled 2169 pages. It's been several months and each week it crawls roughly the same number of pages. Any idea why I'm not getting fully crawled?
-
The best and most efficient solution with any question regarding Moz crawls or tools is to go directly to them with questions (sometimes there's enough unique behind the scenes stuff going on that forum people don't have access to). Their team is great and has helped me a handful of times, quickly and politely.
-
Hi Jack,
Server errors. Redirect loops. Disallowed pages in robots.txt. etc.
Here are some suggestions from www.seomoz.org/help/crawl-diagnostics:
"Why didn’t you crawl all my pages? I only got a one page crawl. Looks like you missed a bunch!
If you suspect you didn’t get a full crawl, or Rogerbot missed some of your pages, there could be several reasons why this happens.
- We only crawl a maximum of 400 links per page. If several pages of your site all have the same 400 links on each page, we may not discover all the pages on your site. Try optimizing your navigation to reduce the number of links.
- Does your navigation rely on JavaScript? Can visitors navigate your site with JavaScript disabled? SEOmoz doesn’t crawl JavaScript, so make sure your links work in all browsing environments.
- Does your site consist of multiple subdomains? Crawls are restricted to the subdomain you set your campaign up on. This means that in general, we don't crawl multiple subdomains. You can solve this by specifying a “Root Domain” crawl in the setup process. (This requires starting a new campaign.)"
If you are certain you don't have any of the above problems, then I would suggest contacting help@seomoz.org
Good luck.
Mike
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to Properly Set Up My Campaign
My url www.pcf.org redirects to ttp://www.pcf.org/site/c.leJRIROrEpH/b.5699537/k.BEF4/Home.htm because of my old CMS and this cannot be changed. Should I set it up as a Subdomain, root domain, or folder? For example using the homepage url http://www.pcf.org/site/c.leJRIROrEpH/b.5699537/k.BEF4/Home.htm I get the following results: subdomain: You've decided to set up a subdomain campaign, but entered the subfolder path:
Moz Pro | | mstanwyck
www.pcf.org/site/c.leJRIROrEpH/b.5699537/k.BEF4. Don't worry, we'll switch that for you and crawl everything on your subdomain: www.pcf.org. If you meant to set this up to only crawl pages in the subfolder /site/c.leJRIROrEpH/b.5699537/k.BEF4, click "Go back and Change" and choose the subfolder option in step 1. root domain: You've decided to set up a root domain campaign, but entered the subfolder path: www.pcf.org/site/c.leJRIROrEpH/b.5699537/k.BEF4. Don't worry, we'll switch that for you and crawl everything in the subfolder /site/c.leJRIROrEpH/b.5699537/k.BEF4. subfolder: It looks like you submitted a URL that relates to an individual file: www.pcf.org/site/c.leJRIROrEpH/b.5699537/k.BEF4/Home.htm
rather than a folder:
www.pcf.org/site/c.leJRIROrEpH/b.5699537/k.BEF4. Don’t worry, we’ll switch that for you and crawl everything in that folder instead. If you meant to set this up for a different folder, click ”Go back and change” and enter a different path. To complicate things further, as you move through the site, the only part of the URL that starys consistent 100% of the time is: http://www.pcf.org/site/c.leJRIROrEpH/ Depending on where you are on the site, it appends the url. For example: One page is http://www.pcf.org/site/c.leJRIROrEpH/b.6185669/k.9792/Sherry_Galloway.htm while another is http://www.pcf.org/site/c.leJRIROrEpH/b.5800843/k.94DE/Everyday_Heroes.htm0 -
Confused on setting up my domain in a campaign
I have a website that is www.example.com. When I first setup my campaign in SEOMoz, I missed the instructions that stated I need to set up my domain as it appeared in the browser. Thus, I set it up as example.com. When I realized my mistake, I setup a second campaign as www.example.com and am running them side by side to see the differences. My question has to do with the best setup. I have a blog that is a subdomain as blog.example.com. It appears that the www.example.com campaign is not crawling the blog pages but example.com is. Are there any downsides in setting up this campaign as example.com? I have a number of other sites with the same issue so would like to get this resolved before I setup the additional sites. Thanks for any assistance or insights into ways I may be "shooting myself in the foot" with the wrong domain structure in my campaign settings.
Moz Pro | | rfwood0 -
Campaign set-up: root domain vs. subdomain
Hi, I have a question about the proper way to set up a new SEOmoz campaign for a simple 5-page website (home, about, services, blog, contact). If I'm tracking only www.domain.com, does it make a difference if I set it up to track just that subdomain or should I track it at the root domain? Is it better to use the root domain? Are there pros or cons to using one over the other? If the website is expanded in the future, any future pages will most likely be on the same subdomain (e.g., www.domain.com/page). Thanks in advance! Carolina
Moz Pro | | csmm0 -
SEOMOZ Crawling Our Site
Hi there, We get a report from SEOMOZ every week which shows our performance within search. I noticed for our website www.unifor.com.au that it looks through over 10,000 pages, however our website sells less than 500 products so not sure why or how so many pages are trawled? If someone could let me know that would be great. It uses up a lot of bandwidth doing each of these searches so if the amount of pages being trawled reduced it would definitely assist. Thanks, Geoff
Moz Pro | | BeerCartel750 -
Drop in number of Pages crawled by Moz crawler
What would cause a sudden drop in the number of pages crawled/accessed by the Moz crawler? The site has about 600 pages of content. We have multiple campaigns set up in our Pro account to track different keyword campaigns- but all for the same domain. Some show 600+ pages accessed, while others only access 7 pages for the same domain. What could be causing these issues?
Moz Pro | | AllaO0 -
Why is the SEOmoz crawler crawling the old version of our website?
Hello, I'm a new SEOmoz member. On Dec. 2nd, after completely redesigning our website, we migrated to a new hosting company by switching our DNS to the new server. The vast majority of the URLs have changed and we configured redirects of the old URLs to the new ones. Although, this task is not completed yet. After the migration, I created an account on SEOmoz to be able to track our progress and find the issues to fix to optimize our SEO. For some reason, in the SEOmoz reports it is the old URLs that show up. Unless the crawler does not actually crawl the pages and only uses the indexed pages to generate its report, I don't understand how could this possible. Anyone has a clue? When will the new URLs be indexed by SEOmoz and the major search engines? Thanks for your help!
Moz Pro | | Gestisoft-Qc0 -
Only 1 page is being crawled by SEOmoz for the last 2 crawls
I would like to ask for the possible problem plus solution on one of our campaigns. Only 1 page is being crawled by SEOmoz for the last 2 crawls. Before the last two crawls, SEOmoz crawls numerous pages and we can’t think of a possible reason for this error. For this particular campaign , there are no data --- no errors, warnings and notices. Thanks!
Moz Pro | | TheNorthernOffice790 -
Campaignes for sub domains
Rand yoou mentioned soon we will be able to list the root domain and be able to track subdomains also in the same campaigne. I could not find the article again now, do you have a ETA?
Moz Pro | | AlanMosley0