MOZ Crawler only crawling one page per campaign
-
We set up some new campaigns, and now for the last two weekly crawls, the crawler is only accessing one page per campaign. Any ideas why this is happening? PS - two weeks back we did "upgrade" the account. Could this have been an issue?
-
Thanks for the suggestions. There was not a robots.txt file. I'm hoping that will do it.
-
Also check out this help desk article on reasons why the crawler gets stuck at one page on a campaign. http://seomoz.zendesk.com/entries/409821-why-isn-t-my-site-being-crawled-you-only-crawled-one-page
-
This happened to me in the past. My client's site had a custom 404 that redirect all missing pages to the homepage. When roger bot (seomoz crawler) searched for the robots.txt page (that was non-existent) it continually got redirect to the homepage.
So on that note, check if you have a robots.txt file. If not, add one and see if it solves the problem.
-
can we see urls, there may be a techniocal reason
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Rogerbot's crawl behaviour vs google spiders and other crawlers - disparate results have me confused.
I'm curious as to how accurately rogerbot replicates google's searchbot I've currently got a site which is reporting over 200 pages of duplicate/titles content in moz tools. The pages in question are all session IDs and have been blocked in the robot.txt (about 3 weeks ago), however the errors are still appearing. I've also crawled the page using screaming frog SEO spider. According to Screaming Frog, the offending pages have been blocked and are not being crawled. Webmaster tools is also reporting no crawl errors. Is there something I'm missing here? Why would I receive such different results. Which one's should I trust? Does rogerbot ignore robot.txt? Any suggestions would be appreciated.
Moz Pro | | KJDMedia0 -
The pages that add robots as noindex will Crawl and marked as duplicate page content on seo moz ?
When we marked a page as noindex with robots like {<meta name="<a class="attribute-value">robots</a>" content="<a class="attribute-value">noindex</a>" />} will crawl and marked as duplicate page content(Its already a duplicate page content within the site. ie, Two links pointing to the same page).So we are mentioning both the links no need to index on SE.But after we made this and crawl reports have no change like it tooks the duplicate with noindex marked pages too. Please help to solve this problem.
Moz Pro | | trixmediainc0 -
How can i get seomoz to crawl a campaign on demand
hi how can i get seomoz to crawl a campaign on demand instead of on a weekly basis? For example i have corrected some error warnings and on page elements and would like it to re crawl the site sooner to see how the corrections have worked? thanks
Moz Pro | | Bristolweb0 -
My Campaign has been crawling for about a week now
Can anyone tell me why one of my campaigns has been stuck in crawl mode for about a full week and it is still not done?!?!
Moz Pro | | nazmiyal0 -
Links & page authority crawl
I see the links and page authority have not been updated in over a month... does anyone know how often it gets updated?
Moz Pro | | nazmiyal0 -
How long does it take for a campaign website crawl to be completed?
Our campaign website crawl has been 'crawling' now for 5 days. Is this a normal phenomenon or is something hanging up?
Moz Pro | | Discountvc0 -
My campaigns are not analyzing all my pages.
Hi I created a campaign against http://www.universalpr.com, and this campaign reports that only one page has been crawled. This site uses a jsvascript redirect to the real page which can be found through the following: www.universalpr.com/wps/portal/universal/univhome/!ut/p/c5/04_SB8K8xLLM9MSSzPy8xBz9CP0os_hQdwtfCydDRwN_Jw9LA0-LAOPQYCdDI_9QY_1wkA6zeAMcwNFA388jPzdVPzi1WL8gO68cANNcdLU!/dl3/d3/L2dBISEvZ0FBIS9nQSEh/ Now I also attempted to create a campaign against this page in case that the javascript redirect was breaking things, but that campaign also reported 1 page crawled. Can anyone instruct me as to what I'm doing wrong? Thank you
Moz Pro | | jcmoreno0 -
On-Page Optimisation tool on intranet pages
Does anybody know if there's any easy way to use the On-Page Optimisation tool on intranet or not publicly accessible pages? Thanks!
Moz Pro | | neooptic0