Crawl test from tools
-
Hi,
I notice that the crawl test which is from the Research Tools doesn't really get a new crawl even though there is 2 crawl per day. It will only provide the data which was acquire from the crawl diagnostics in my pro account. There is no point for me to get the data which I get from my crawl diagnostic isn't it? Even seomoz provided with more than 2 crawl per day also useless in this case.
This whole thing doesn't make sense as the crawl diagnostics will only perform a full crawl test once every week. but even the crawl test also not helping any thing out for me.
-
Everyone, thanks for helping... Greatly appreciated.
-
Hi there Hanzo,
Thanks for reaching out! it looks like you are having some trouble with the crawl test. Keri is correct that our crawl test uses "Cached" information to construct your crawl, which can take up to 2 days to clear. While I am not sure why the sites are still reporting 404s, there's a few things that could explain why you might be getting a stale crawl:
- Your server simply did not update the changes you've made on the backend.
- There might an issue with our crawler, which in my last status there isn't a outage that I know of.
I looked into your crawl data and the 404s seems to be confirmed by the report, if you download your latest report and do a sort you should see that
Hope I was able to provide some insight, please let me know if you have questions!
Best,
-
I believe the crawl test has a cache of a couple of days. I'm asking someone from the help desk to come in and confirm, but it may be Monday before they have a chance to do so.
-
I know there is third party app which I can use to crawl,
but then isn't the crawl test is there for us to do testing instead of using other app rather than seomoz's tool?
-
You may want to give screaming frog or xenu's link sleuth. They should be able to crawl you site and give you the info you need.
-
From what I do with the crawl test, it doesn't show me the latest crawl. I need to check whether all the error 404 or related error was being solved. It only show me the old data which was crawl before but not the latest one.
-
Hanzo, what is it that you need to do that the crawl test isn't helping you with?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Could my Crawl Error Report be wrong?
HI there, I am using Yoast SEO plugin on a wordpress website. I am currently showing 70 Med priority crawl errors 'missing meta description' on my Moz pro account. This number of missing meta descriptions has increased over the last 6 weeks. But every single page / post / tag / category has both the title and meta description completed via Yoast. I requested a google bot to crawl the site a few weeks ago as thought it perhaps wasn't been crawled and updated. Any idea what the issue might be? Could the Moz report be incorrect for some reason? Or could something be blocking Moz / Google from seeing the Yoast plugin?
Moz Pro | | skehoe0 -
OK Crawl Test Link Question Again!
I've downloaded a crawl test and column G Link Count reads 62 and yep there are a total of 62 links on the page in question. Column AM Internal Links reads 303 and yep there are somewhere in the order of 303 pages pointing at this one. Root Domains is surprisingly low at 6, so maybe there are only 6 domains linking to this page. BUT... External Links read 51. There are not 51 links pointing away from this domain on this page, no way hozay, so can anybody tell me what is meant by 'External Links? A humble thank you in anticipation of an education. Jem
Moz Pro | | JemRobinson0 -
Not all pages are being crawled
I am set up on the PRO plan, I was under the impression that it would crawl up to 10,000 pages. My site has just over 200 pages, but whenever I am crawled it only crawls 121 pages. Is this normal? It's hard to know how reliable my data is because a significant amount of pages are missing.
Moz Pro | | KristinHarding0 -
Tools that crawl 2 million page sites
Our site is about 2million pages deep, 50% of which is stale content. Yes, I know - OMG #unhygienic. Even if we get approval to get rid of half of it. SEOMoz Pro Elite only crawls 20k deep - what can i do to crawl and diagnose the whole site. Are there any tools anyone can suggest. SEOMoz??
Moz Pro | | ilhaam0 -
Competitive Link Analysis Tool?
Hi, I ran a competitive link analysis report today and back came quite a few domains that 2 or more of my 5 main competitors link from. Is it worth me submitting links to these sites? And would i be best served submitting my homepage URL or submitting a brand page such as Creative Recreation Trainers? I want to target that brand but don't want to do it if my main URL is better? Any ideas? See below my report. | Subdomain | Subdomain mR | Subdomain mT | # Competitors | # Linking Pages | Link Acquired |
Moz Pro | | YNWA
| t.co/ | 8.05 | 8.04 | 2 | <a>2</a> | |
| ww2.cox.com/ | 5.99 | 6.50 | 2 | <a>3</a> | |
| www.littlewebdirectory.com/ | 5.90 | 5.59 | 2 | <a>2</a> | |
| www.amazines.com/ | 5.69 | 5.66 | 2 | <a>3</a> | |
| svpply.com/ | 5.66 | 5.53 | 3 | <a>20</a> | |
| www.jayde.com/ | 5.64 | 5.68 | 3 | <a>4</a> | |
| www.pearltrees.com/ | 5.58 | 5.81 | 2 | <a>2</a> | |
| www.businessseek.biz/ | 5.52 | 5.51 | 2 | <a>3</a> | |
| www.a1articles.com/ | 5.50 | 5.22 | 3 | <a>9</a> | |
| www.linksilo.de/ | 5.48 | 5.23 | 2 | <a>15</a> | |
| www.alistsites.com/ | 5.46 | 5.24 | 2 | <a>38</a> | |
| www.the-free-directory.co.uk/ | 5.37 | 5.07 | 2 | <a>20</a> | |
| www.walhello.com/ | 5.30 | 4.97 | 2 | <a>2</a> | |
| www.quarkbase.com/ | 5.14 | 5.12 | 2 | <a>2</a> | |
| snipsly.com/ | 5.13 | 5.20 | 2 | <a>21</a> | |
| www.counterdeal.com/ | 5.12 | 5.07 | 2 | <a>2</a> | |
| www.01webdirectory.com/ | 5.03 | 5.03 | 2 | <a>2</a> | |
| www.2addlink.info/ | 4.92 | 4.58 | 3 | <a>4</a> | |
| www.fuk.co.uk/ | 4.64 | 5.00 | 3 | <a>20</a> | |
| www.final-fantasy.us/ | 4.63 | 4.77 | 2 | <a>2</a> | |
| oyax.com/ | 4.42 | 4.61 | 2 | <a>4</a> | |
| www.touchretail.co.uk/ | 4.33 | 4.21 | 2 | <a>4</a> | |
| tptbtv.cold10.com/ | 4.27 | 4.86 | 3 | <a>1</a> | |
| www.mastbusiness.com/ | 4.23 | 4.34 | 2 | <a>2</a> | |
| www.competitionhunter.com/ | 4.16 | 4.21 | 2 | <a>6</a> | |0 -
Seomoz crawling filtered pages
Hi, I just checked an seo campaign we started last week, so I opened seomoz to see the crawl diagnostics. Lot's of duplicate content & duplicate titles showing up, but that's because Rogerbot is crawling all of the filtered pages as well. How do I exclude these pages from being crawled? /product/brand-x/3969?order=brand&sortorder=ASC
Moz Pro | | nvs.nim
/product/brand-x/3969?order=popular&sortorder=ASC
/product/brand-x/3969?order=popular&sortorder=DESC&page=10
/product/brand-x/3969?order=popular&sortorder=DESC&page=110 -
Why Is SEOMOZ No Longer crawling All Of My Site
Hi all, I joined Seomoz over a month ago and Roger has been crawling all of the pages on the site approx 20 pages. Through out the last few weeks I have been working on the errors and notices identified by Roger. However, this week Roger has only re-crawled 1 page and is not picking up all the other pages. Has any one come across this problem. can you recommend any thing to resolve it? Many thanks in advance....
Moz Pro | | Dan280 -
Clearing the Tool Reports You've Run
Is there anyway of clearing the 'Tool Reports You've Run' list on the Pro Dashboard, or setting certain reports not to be saved here?
Moz Pro | | BigMiniMan0