How does SEOMoz crawl sites? Does it follow the sitemap?
-
I only ask because in my report it returned a few "no follow" links. Was wondering how SEOMoz operates.
-
Rogerbot, our crawler, obeys the robots.txt nofollow rules associated with your site. For some reason your robots.txt file is blocking our crawler from seeing those pages. For more information on how the tool works, you can view this incredible resource: http://www.seomoz.org/help/rogerbot-crawler that explains how our crawler works in vivid detail. I hope this helps!!
-
My guess is they spider the way Googlebot spiders - finding links on your home page, following those links, and finding more links on the subsequent pages, so on and so forth. I doubt they look at the sitemap, since it rarely reflects the true importance of pages on your site, which tend to be linked to more often.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Crawl 4xx Errors?
Hello! When I check our website's critical crawler issues with Moz Site Crawler, I'm seeing over 1000 pages with a 4xx error. All of the pages that are showing to have a 4xx error appear to be the brand and product pages we have on our website, but with /URL at the end of each permalink. For example, we have a page on our site for a brand called Davinci. The URL is https://kannakart.com/davinci/. In the site crawler, I'm seeing the 4xx for this URL: https://kannakart.com/davinci/URL. Could this be a plugin on our site that is generating these URLs? If they're going to be an issue, I'd like to remove them. However, I'm not sure exactly where to begin. Thanks in advance for the help, -Andrew
Moz Pro | | mostcg0 -
Missing Title for Sitemap
Our site is built on Wordpress and we use a very popular SEO plugin called Yoast to generate our sitemap (as well as handle multiple other SEO functions). When MOZ's spider crawls our site, this sitemap triggers an error saying "Missing Title or Empty." My question is how can I avoid having this error hurt me in terms of my rankings. It seems strange to me that such a ubiquitous plugin would be generating something as important as a sitemap in an incorrect format.
Moz Pro | | ShatterBuggy0 -
How to make a Crawl Report readable?
Hi! I am trying to find out how I make my CSV report neat, so I can interprete the report. I know have a CSV report with just numbers and text all in one column. I tried the button text to columns but that doesn't work because when I do that at column A it overwrites column B in which I have the same problem! Thanks
Moz Pro | | HetCommunicatielokaal0 -
How to creat link building,why seomoz has no such tool? also why it not crawl on daily basis?
how to creat link building? why seomoz has no such tool? also why it not crawl on daily basis?
Moz Pro | | mrgunii0 -
20000 site errors and 10000 pages crawled.
I have recently built an e-commerce website for the company I work at. Its built on opencart. Say for example we have a chair for sale. The url will be: www.domain.com/best-offers/cool-chair Thats fine, seomoz is crawling them all fine and reporting any errors under them url great. On each product listing we have several options and zoom options (allows the user to zoom in to the image to get a more detailed look). When a different zoom type is selected it adds on to the url, so for example: www.domain.com/best-offers/cool-chair?zoom=1 and there are 3 different zoom types. So effectively its taking for urls as different when in fact they are all one url. and Seomoz has interpreted it this way, and crawled 10000 pages(it thinks exist because of this) and thrown up 20000 errors. Does anyone have any idea how to solve this?
Moz Pro | | CompleteOffice0 -
Yahoo Site Explorer died. Does DA and PA Seomoz parameters have all it takes to replace YSE?
We're trying to simplify our link analysis job when acquiring a new external link. We're wondering to know if Domain Authority Seomoz parameter is all what we need to take a decision. Does DA Seomoz parameter takes into consideration 'domain age', 'alexa ranking', 'dmoz links', indexed pages on google', and 'inbound links vs outbound links'?
Moz Pro | | seoneogames1 -
Campaign Not Crawling
I set up my first 5 campaigns and one is not crawling beyond one-page. It's been over 48 hours. This site has nearly 3.500 pages the others much less, however, this shouldn't make any difference. I searched for the problem and couldn't find it so I hope this question isn't redundant. Comments and advice would be appreciated.
Moz Pro | | JavaManOne0 -
Dismiss crawl diagnostics error
Hello everyone, Is there a way to dismiss some errors in the Crawl Diagnostics tool so they don't appear again? It happens so that some of the errors are never going to be fixed because of their nature. For example, 'Title too long' errors that point to some of the threads on my forum - it doesn't make sense to change the title of a thread posted by user just for the sake of the error disappearing from the 'Crawl Diagnostics' tool. 🙂 Otherwise the CD interface gets a little bit cluttered with errors which I will never fix anyway. I wonder how others deal with this problem. Thanks.
Moz Pro | | MaratM0