Does anyone know of a crawler similar to SEOmoz's RogerBot?
-
As you probably know SEOmoz had some hosting and server issues recently, and this came at a terrible time for me... We are in the middle of battling some duplicate content and crawl errors and need to get a fresh crawl of some sites to test things out before we are hit with the big one?
Before I get a million thumbs downs- I love and will continue to use SEOmoz, just need something to get me through this week ( or until Roger is back! )!
-
There are a bunch to choose from, but here's a list of some freebies that pack a big punch:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Missing/Duplicate Content but it's definitely all right!
We are having some issues flag up through our MOZ software on multiple pages on www.ukpods.co.uk website such as missing content, missing metatags, missing title, duplicate content, Missing/Invalid H1 but having checked and spoken to our website developer all the content is there. 1 page as an example is https://ukpods.co.uk/pods-storage/decluttering/ It says the below items are missing or invalid but you cna see they are all there in the below - are they not being picked up and read for any reason? Title: http://bit.ly/2P6v9To Description: http://bit.ly/356BkvY H1: http://bit.ly/2PrlmpU Canonical tag: http://bit.ly/355rQRU Any help would be much appreciated. Thanks!
Moz Pro | | RachelMiller0 -
I have an issue with hubspot's blog platform and duplicate content.
It is redirecting all https to http and I am unable to change it, resulting in a lot of duplicate content. Has anyone else experienced this? If so, did you find a solution? Or does anyone have any suggestions?
Moz Pro | | KurtzGro0 -
Are the metrics in Moz's SERP analysis relevant in my area?
Our site (https://loco2.com/) specialises in train journey planning at ticket booking, so we want to improve our ranking when people search Google for a specific journey they want to make (eg "London to Barcelona by train"). We have a set of about 400 Guide pages (eg London to Barcelona) which we want people to find when they search for those journeys. I've been trying to find out why some of those pages do better than others in search, and what our competitors are doing better. I was hoping that the SERP analysis and Keyword Reports tool could help me with this, so I ran analyses for a set of the keyword phrases we want to rank for, but I was surprised to find that most of the metrics in both the basic and full reports don't seem to relate to how high in the search result a page ranks. The screengrab attached shows the results for "Brussels to Paris train", which are fairly typical - many of the top-ranking pages have fairly low scores for most metrics, and the metrics don't seem to have much effect on how high they are in the search results. ( I assume N/A means the Page Authority is very low, or is it not calculated for some other reason?) I looked into this a bit further by downloading full reports for 11 keyword searches, and comparing the average scores for each metric with the rank, but I still couldn't see any effects of the metrics on the ranks. If anything, pages with higher Page Authority and Domain Authority were further down in search rankings. Does this mean that link metrics aren't very important for the kinds of pages we're looking at? This makes sense to me, because if I'm searching for tickets from London to Barcelona I want to find a very specific results page which is unlikely to have many links, but I keep reading that link metrics are one of the most important factors in SEO. Am I misunderstanding something? wJ9uUcG
Moz Pro | | MargotLoco20 -
'Too many on-page links': doesn't add up...
Hi All, I'm fairly new to the site, have had a full site crawl and am somewhat confused by a large number of pages (82) reported as showing 'too many on page links'. Mine is a new e-commerce site selling a project management methodology. I have about 80 pages which describe individual templates and contain internal links to additional, related content on the site. The crawl warnings define too many on page links as roughly 100 "on any given page" yet almost none of the pages flagged contain more than 20-30 on links. Any thoughts and ideas about (a) the degree to which this will aeffect my ranking in practice and (b) how to resolve it gratefully received! Thanks in advance, Felix
Moz Pro | | RomanCat0 -
How to read Crawler downloaded report
I am trying to seperate the duplicate title and description URLs, by looking at the report i am not getting how to find all urls which contain same title and description. Is there any video link on the site which walk me through each part of the report. Thanks, Punam
Moz Pro | | nonlinearcreations0 -
SEOMoz Crawling Only 1 Page
I entered a new site into my dashboard 2 days ago - everything looked kosher, there were a few hundred pages crawled and a whole bunch of errors. I came back this morning to start work on the site and SEOMoz has crawled the site again, this time returning only 1 page and 0 errors. I haven't even logged in to the site since the first crawl, so I couldn't have broken anything. Has anyone seen this before?
Moz Pro | | Junction0 -
Discrepancies in link reports from SEOMoz and OSE
Hello, I am trying to understand the link profile of my site. In the competitive analysis from SEOMoz, it shows 22K+ of links, with about 900+ being external followed links and 800+ internal. Webmaster tools shows a similar number of external followed links, but the link report from OSE only shows about 150 external followed links. This brings up several questions: Where is this 22K total links coming from? Why is there such a difference in the SEOMoz competitive analysis and the OSE report? Why does the OSE report show some links as not followed, while Webmaster tools lists them as followed? Am new to the off-page aspects of SEO, and I have to say the discrepancy in data from different tools is by far the most challenging thing for me. Would welcome general advice on this topic as well. Thank you!
Moz Pro | | LynnMarie0 -
Blocking all robots except rogerbot
I'm in the process of working with a site under development and wish to run the SEOmoz crawl test before we launch it publicly. Unfortunately rogerbot is reluctant to crawl the site. I've set my robots.txt to disallow all bots besides rogerbot. Currently looks like this: User-agent: * Disallow: / User-agent: rogerbot Disallow: All pages within the site are meta tagged index,follow. Crawl report says: Search Engine blocked by robots.txt Yes Am I missing something here?
Moz Pro | | ignician0
