MOZ Crawler
-
Hi,
how much time it will take MOZ crawler to take entire site? In 24 hours it crawled only 500 pages isn't it too slow? My website has almost 50k pages.
-
Hi there,
Sam from Moz's Help Team here! Could you pop a message over to help@moz.com about this so we can ensure everything is running smoothly for you?
Thank you!
-
Yeah it will take a while but it's necessarily slow. Moz's crawler has to share bandwidth with your real, human users. If they crawl your site too fast they could lag your site or accidentally DDoS you or something
Since it's a paid for service, I'm guessing that Moz err on the side of caution with Rogerbot and don't crawl insanely fast. I know that Rogerbot supports adding a crawl delay to make the crawling even slower:
https://moz.com/help/moz-procedures/crawlers/rogerbot
But I don't think there are any 'go faster' commands. Whilst I really like Moz's crawler I tend to find that the SEMRush one is faster for on-site stuff, though it does have URL crawling caps
To be honest, just get Screaming Frog and learn how to use that. Now that it has a database storage mode (instead of maniacally trying to store all the crawl data in the RAM) - it probably could crawl all 50K URLs. It could probably do it faster than Rogerbot, though you wouldn't wanna go nutty as you could easily DDoS yourself :')
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Domain Authority 27
Hi there, I have been reading up about Domain Authority being important to rank though I have a question about it. When trying a few google searches and using a tool to see the Moz Domain Authority i notice the following. I won't mention sites or keywords, happy to send them via private message if required. A site ranks 1st position for most keywords including the main keyword that related to their industry, a highly competitive keyword. Their Moz Domain Authority 27, Est. Visits: 265,000, Pinterest shares 72, Facebook shares 191 and unique domains that point to it are 55. Yet when I look at another site, optimising for the same keyword, Moz Domain Authority 50, Est. Visits: 60,000, Pinterest shares 35000, Facebook shares 20000 and unique domains that point to it are 521. I would have thought that the second site ranks 1st for the keyword and the first one mentioned which is in 1st position I would have expected to be on page two or three. as I scroll down the first page for the term, all sites except for one have higher DA, visits, shares and so on. If a high PA is required can someone help me to understand the above example? Whn a low PA site ranks much better for a primary keyword than all other sites that have a higher PA than it?
Moz Pro | | IvanaDaulay1 -
Moz can't see my link from an external site
Hi, I'm hoping someone can shed some light on this. I've had a followed link created from http://www.dubaisharks.com/ pointing to my website - http://www.yallarugby.com but in the reports it doesn't show that there is a link coming fro this website. The link has been active for a few months now, but still no record of it. Any ideas?
Moz Pro | | SeoSheikh0 -
Meta Tag Descriptions not being found in Moz Crawls
Hey guys, I have been managing a few websites and have input them into Moz for crawl reports, etc. For a while I have noticed that we were getting a gratuitous amount of errors when it came to the number of missing meta tags. It was numbering in the 200's. The sites were in place before I got here and a lot of the older posts no one had even attempted to include tags, links of the page or anything. As they are all Wordpress Sites and they all already had the Yoast/Wordpress SEO plug-in installed on them, I decided I would go through each post and media file one at a time and update their meta tags via the plug in. I personally did this so I know that I added and saved each one, however the Moz crawl reports continue to show that we are missing roughly 200 meta tags. I've seen a huge drop off in 404 errors and stuff since I went through and double checked everything on the sites, however the meta tag errors persist. Is this the case that Moz is not recognizing the tags when it crawls because I used the Yoast Plugin? Or would you say that the plugin is the issue and I should find another way to add meta tags to the pages and posts on the site? My main concern is that if Moz is having issues crawling the sites, is Google also seeing the same thing? The URLS include:
Moz Pro | | MOZ.info
sundancevacationsblog.com
sundancevacationsnews.com
sundancevacationscharities.com Any help would be appreciated!0 -
Has the Hummingbird Changes Effected the accuracy of MOZ?
Hello, I am new to the forum. Can anyone tell me if the recent Hummingbird Changes have effected the accuracy of MOZ? Specifically, keyword comparison of competition. Thanks, Derek
Moz Pro | | Chris81980 -
Not provided traffic in Moz
Hello, I was wondering, if anyone knew how moz handles non provided keywords? is it not displayed, or is it labeled as something else? I know in WMT, it still displays it. Thanks for your help.
Moz Pro | | PeterRota1 -
Moz scores
Hi All, Could anyone please explain for me why the moz scores that show under SERPs in Google search show one set of values, but when i click on the SERP listing and go to that page, the Moz scores at the bottom of the tab the page is open in are completely different. I.E - SERP result Wireless Alarms <cite>www.wireless-alarms.net/</cite>Honeywell, Friedland, Response, Wireless house home intruder burglar alarms and wirefree doorbells chimes and accessories.<iframe class="SEOmoz-iframe" src="chrome-extension://eakacpaijcpapndcfffdgphdiccmpknp/html/serpbar.html#{" settings":{"dmr":false,"dmt":false,"mr":true,"mt":true,"serp-overlay":true,"toolbar-enabled":true,"subdomain-metrics":false,"serp-panel-open":true,"toolbar-position":"bottom"},"lsdata":{"feid":3,"fipl":3,"fmrp":2.0446037910058426,"fmrr":5.396862303226778e-12,"ftrp":1.8723654310590483,"ftrr":9.21604483751679e-12,"fuid":3,"pda":8.903939958502189,"peid":3,"pid":3,"pmrp":1.6026230610187469,"pmrr":1.3566326316606153e-10,"ptrp":1.025503845909542,"ptrr":1.4972548621304158e-11,"puid":3,"ueid":2,"uemrp":2.577887966559377,"uemrr":5.40110633420739e-13,"uid":2,"uipl":2,"ujid":2,"umrp":2.577887966559377,"umrr":5.40110633420739e-13,"upa":23.634692786039714,"utrp":4.9331532541404854,"utrr":1.9684091945588755e-16},"user":{"level":"pro","options":{},"proenabled":true,"aclurl":"http:="" moz.com="" users="" level?src="mozbar","loginURL":"https://moz.com/login","logoutURL":"https://moz.com/logout","apiRequest":"url-metrics","cols":"133146078688","expired":false,"accessId":"pro-RGFkZHlTbXVyZg%3D%3D","expires":1374925378,"signature":"tBSBXpLbBmk073ofsfBwnilmODw%3D","displayName":"DaddySmurf"},"serp":4,"host":"www.wireless-alarms.net"}"" scrolling="no"></iframe> Page result <label>Page:</label><label id="Page Authority" class="key lsdata">PA:</label> <label id="SEOmoz-data-PA" class="val lsdata">27</label><label id="mozrank" class="key lsdata" title="MozRank">mR:</label> <label id="SEOmoz-data-mR" class="val lsdata">5.30</label><label id="moztrust" class="key pro-only lsdata" title="MozTrust">mT:</label> <label id="SEOmoz-data-mT" class="val pro-only lsdata">5.61</label><label id="SEOmoz-data-uid">1,950</label> links from
Moz Pro | | DaddySmurf
<label id="SEOmoz-data-uipl">2</label> Root Domains<label>Root Domain:</label><label id="dom-pageauthority" class="key lsdata" title="Domain Authority">DA:</label> <label id="SEOmoz-data-DA" class="val lsdata">24</label><label id="SEOmoz-data-puid">100,262</label> links from
<label id="SEOmoz-data-pid">3</label> Domains<label>Subdomain:</label><label id="SEOmoz-data-fuid">19,953</label> links from
<label id="SEOmoz-data-fipl">3</label> DomainsThank you.Si0 -
Moz is officially demigod-approved!
So, I was reading Percy Jackson & the Olympians: The Battle of the Labyrinth tonight, and found a description of a graffiti tag in the Labyrinth that says "MOZ RULZ". What do you think? Is one of the Mozzers a demigod who has explored and survived the Labyrinth? 🙂 (See attachment for book quote.) F1WmT5r
Moz Pro | | AdamThompson4 -
Drop in number of Pages crawled by Moz crawler
What would cause a sudden drop in the number of pages crawled/accessed by the Moz crawler? The site has about 600 pages of content. We have multiple campaigns set up in our Pro account to track different keyword campaigns- but all for the same domain. Some show 600+ pages accessed, while others only access 7 pages for the same domain. What could be causing these issues?
Moz Pro | | AllaO0