How much time should I wait between Crawl Tests?
-
Hello!
I ask because it has happened before (and again this morning) that after doing a crawl test and repairing my site per the errors found in Moz's crawl test it still finds the same error. Even though I fixed them.
Typically I do a re-crawl 6 hours after or the next day and I find the same errors. I know they are fixed because a couple of days go by and finally Moz gets it right.
I had understood that the crawl test was an "on-demand" crawl of sorts, granted with limit of 2 a day. But it seems that if you re-crawl your site within a day the same results yield? It's frustrating.
Is this correct?
Thank you!
-
Hi Erin, thank you for your response. Ok that makes sense.
I appreciate it!
-
Ahoy!
Erin here. I'm a lead on the Help Team here at Moz! Sorry for the confusion! Our Crawl Test data is actually cached for 48 hours, which is why you're seeing the same issues after 6 hours/same day after you've made fixes. In order to get the most accurate results, you'll want to wait at least 48 hours between Crawl Tests.
I hope this helps, and if you have any other questions, feel free to shoot us an email at help@moz.com!
Yours til the Chocolate Chips,
Erin
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl tests stuck in queue
I have tried to run a number of crawl tests recently for our client's sites outside the US and they have been stuck in the queue for over a week. 3 of them completed, but then 5 are stuck. Anyone experience this? I haven't seen anything about crawl tests having issues right now.
Moz Bar | | rmcgrath810 -
Canonical in Moz crawl report
I'm wondering if the moz bot is seeing my rel="canonical" on my pages. There are 2 notices that are bothering me: Overly Dynamic URL Rel Canonical Overly Dynamic URL - This notice is being generated by urls with query strings. On the main page I have the rel="canonical" tag in the header. So every page with the query string has the canonical tag that points to the page that should be indexed. So my question...Why the notice? Isn't this being handled properly with the canonical tag? I know I can use my robots.txt or the tool in Google search console but is it really necessary when I have the canonical on every page? Here is one of the links that has the "Overly Dynamic URL" notice, as you can see the the canonical in the header points to the page without the query string: https://www.vistex.com/services/training/traditional-classroom/registration-form/?values=true&course-title=DMP101 – Data Maintenance Pricing – Business Processes&date=March 14, 2016 Rel Canonical - Every page in my report has this notice "Using rel=canonical suggests to search engines which URL should be seen as canonical". I'm using the rel="canonical" tag on all of my pages by default. Is the report suggesting that I don't do this? Or is it suggesting that I should? Again...why the notice?
Moz Bar | | Brando160 -
Weird 404 in Crawl Diagnostics
I'am getting a lot of 404 errors (196 to be precise ) - but their pattern is weird.
Moz Bar | | oorbo
The page that the crawler is trying to find is (e.g):
http://www.oorbo.com/item/asufa-israeli-design-shop**/www.oorbo.com.
the linking page is** http://www.oorbo.com/item/asufa-israeli-design-shop meaning it adds to the end of the link the root URL - /www.oorbo.com. This happens in all 196 cases - trying to find a page http://www.oorbo.com/some-page/www.oorbo.com from a refferer page http://www.oorbo.com/some-page. Obviously this pages do not exist, and it's getting a 404. I've look into the pages themselves and digged into their code - It doesn't seem that the bad link is any where on the page. Did anyone came across this kind of issue? any one can point me to a solution ?0 -
Odd crawl test issues
Hi all, first post, be gentle... Just signed up for moz with the hope that it, and the learning will help me improve my web traffic. Have managed to get a bit of woe already with one of the sites we have added to the tool. I cannot get the crawl test to do any actual crawling. Ive tried to add the domain three times now but the initial of a few pages (the auto one when you add a domain to pro) will not work for me. Instead of getting a list of problems with the site, i have a list of 18 pages where it says 'Error Code 902: Network Errors Prevented Crawler from Contacting Server'. Being a little puzzled by this, i checked the site myself...no problems. I asked several people in different locations (and countries) to have a go, and no problems for them either. I ran the same site through Raven Tool site auditor and got some results. it crawled a few thousand pages. I ran the site through screaming frog as google bot user agent, and again no issues. I just tried the fetch as Gbot in WMT and all was fine there. I'm very puzzled then as to why moz is having issues with the site but everyone is happy with it. I know the homepage takes 7 seconds to load - caching is off at the moment while we tweak the design - but all the other pages (according to SF) take average of 0.72 seconds to load. The site is a magento one so we have a lengthy robots.txt but that is not causing problems for any of the other services. The robots txt is below. Google Image Crawler Setup User-agent: Googlebot-Image
Moz Bar | | Arropa
Disallow: Crawlers Setup User-agent: * Directories Disallow: /ajax/
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
#Disallow: /js/
#Disallow: /lib/
Disallow: /magento/
#Disallow: /media/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/
Disallow: /catalog/product
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
#Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/ Files Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt Paths (no clean URLs) #Disallow: /.js$
#Disallow: /.css$
Disallow: /.php$
Disallow: /?SID= Pagnation Disallow: /?dir=
Disallow: /&dir=
Disallow: /?mode=
Disallow: /&mode=
Disallow: /?order=
Disallow: /&order=
Disallow: /?p=
Disallow: /&p= If anyone has any suggestions then please i would welcome them, be it with the tool or my robots. As a side note, im aware that we are blocking the individual product pages. Too many products on the site at the moment (250k plus) which manufacturer default descriptions so we have blocked them and are working on getting the category pages and guides listed. In time we will rewrite the most popular products and unblock them as we go Many thanks Carl0 -
Why is 410 (Gone) being classed as a high priority issue in crawl diagnostics?
Are high priority issues have suddenly soared by over 100 because Moz is classing 410s as high priority.
Moz Bar | | Melissabraz
Google doesn't class these as so serious, so we were wondering if anyone knows why Mos does?0 -
Dashboard Stats over time? Annual, month-to-month, comparisons?
I'm looking at our data -- we've been in PRO for about 18-months now -- and I don't see how to compare periods. Seems that the gains and losses are data compared in the same week (which is meaningless).
Moz Bar | | jmueller0823
Anyone? Thanks.1 -
Why does Moz Analytics drop my Google Analytics connection every time I login?
I get the following message every time I check the Moz Analytics dashboard on any of the sites I manage. It seems like a bug in the system. Does anyone know how to fix it? "We lost connection with your Google Analytics account. Don't worry — you won't lose any data. Please reauthorize now."
Moz Bar | | cbizzle0 -
Moz crawl sees meta description but there are none
I have a new site I have run a starter crawl on. The crawl came back saying some of the pages do indeed have meta descriptions. When I go to the same page and use the Moz Chrome toolbar it says they do not have meta descriptions. I also know they do not have meta descriptions. Are there any instances in which the Moz crawler would see them when they are not there?
Moz Bar | | SBXMedia0