Order of urls in SEOMoz crawl report
-
Is there any rhyme or reason to the order of urls in the SEOMoz crawl report, or are the urls just listed in random order?
-
Thanks Matt! Glad to know someone else had this question. It's been a head scratcher for me for a while.
-
Hi Lynn,
I once spoke to an SEOmoz staff member about this, and they told me that there is no logic to the way Roger crawls a website - it's completely random. I have looked through my last 10 or so CSVs and each time I have a different order. The first few links of my latest report are from the second level of architecture, so I would concur that it is at least randomly reported, if not randomly collected.
Matt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
WEbsite cannot be crawled
I have received the following message from MOZ on a few of our websites now Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. I have spoken with our webmaster and they have advised the below: The Robots.txt file is definitely there on all pages and Google is able to crawl for these files. Moz however is having some difficulty with finding the files when there is a particular redirect in place. For example, the page currently redirects from threecounties.co.uk/ to https://www.threecounties.co.uk/ and when this happens, the Moz crawler cannot find the robots.txt on the first URL and this generates the reports you have been receiving. From what I understand, this is a flaw with the Moz software and not something that we could fix form our end. _Going forward, something we could do is remove these rewrite rules to www., but these are useful redirects and removing them would likely have SEO implications. _ Has anyone else had this issue and is there anything we can do to rectify, or should we leave as is?
Moz Pro | | threecounties0 -
5XX (Server Error) on all urls
Hi I created a couple of new campaigns a few days back and waited for the initial crawl to be completed. I have just checked and both are reporting 5XX (Server Error) on all the pages it tried to look at (one site I have 110 of these and the other it only crawled the homepage). This is very odd, I have checked both sites on my local pc, alternative pc and via my windows vps browser which is located in the US (I am in UK) and it all works fine. Any idea what could be the cause of this failure to crawl? I have pasted a few examples from the report | 500 : TimeoutError http://everythingforthegirl.co.uk/index.php/accessories.html 500 1 0 500 : Error http://everythingforthegirl.co.uk/index.php/accessories/bags.html 500 1 0 500 : Error http://everythingforthegirl.co.uk/index.php/accessories/gloves.html 500 1 0 500 : Error http://everythingforthegirl.co.uk/index.php/accessories/purses.html 500 1 0 500 : TimeoutError http://everythingforthegirl.co.uk/index.php/accessories/sunglasses.html | 500 | 1 | 0 | Am extra puzzled why the messages say time out. The server dedicated is 8 core with 32 gb of ram, the pages ping for me in about 1.2 seconds. What is the rogerbot crawler timeout? Many thanks Carl
Moz Pro | | GrumpyCarl0 -
No follow links also been reported in SEOmoz crawl diagnostics
Hi, Why does SEOmoz reports links which has been marked as 'nofollow'. I am getting 'Overly-Dynamic URL' reports on links which I have designated as nofollow which means Google will discount them. So why does SEOmoz still report them. Thanks.
Moz Pro | | malpani0 -
Viewing past campaign reports
This is a basic SEOmoz question - Is there a way to view past campaign reports? I haven't manually "archived" any of my reports, because I didn't realize that may be important so it seems I might be out of luck on this. i would like to compare old reports to current ones and can't find a way to access the old reports.
Moz Pro | | gfiedel0 -
Rankings Different in Serps from Report
My site shows up on page one for many search queries, but I'm "not in the top 50" for those same terms according to the rankings report. Having troubleshooted the problem, it may be a canonical issue. I did not include www when I originally set up my reports and it seems that my site shows up in search results with www. However; if you type in the url without the www, it will automatically forward to the page with www. I did this intentionally in order not to get penalized for duplicate content pages. So here's my question: If I've got one page redirected to the other, shouldn't the rankings report also follow that redirect? If so, then it should show the correct results, right? Thanks, Dino
Moz Pro | | Dino640 -
Video SERP and Seomoz ranking
It seems that my keywords that lead to videoresults in SERP, is not being ranked on Seomoz! How come when i rank #3 on a certain keyword with a videoresult Seomoz is displaying me as #15 (which is the second of my pages in SERP)? What is the reason for not counting the video-result in the ranking-tool?
Moz Pro | | alsvik0 -
SEOmoz Report: Competitive Domain Analysis
Hi there, According to SEOmoz Dashboard's Competitive Domain Analysis, the 'Total Links' for our domain shot up (more than quatripled) in the last few weeks. We haven't really done much link building in over 6 months so not sure how to explain this. Did SEOmoz make updates to their bots? Could it be that the bots are finding all these links only now? We've been using SEOmoz reporting for almost a year now, and I've never seen this before. Some insight / explanation for this would be great. Thanks in advance,
Moz Pro | | RBA
Gemma0 -
Crawl Errors Confusing Me
The SEOMoz crawl tool is telling me that I have a slew of crawl errors on the blog of one domain. All are related to the MSNbot. And related to trackbacks (which we do want to block, right?) and attachments (makes sense to block those, too) ... any idea why these are crawl issues with MSNbot and not Google? My robots.txt is here: http://www.wevegotthekeys.com/robots.txt. Thanks, MJ
Moz Pro | | mjtaylor0