Does anyone know of a crawler similar to SEOmoz's RogerBot?
-
As you probably know SEOmoz had some hosting and server issues recently, and this came at a terrible time for me... We are in the middle of battling some duplicate content and crawl errors and need to get a fresh crawl of some sites to test things out before we are hit with the big one?
Before I get a million thumbs downs- I love and will continue to use SEOmoz, just need something to get me through this week ( or until Roger is back! )!
-
There are a bunch to choose from, but here's a list of some freebies that pack a big punch:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz shows duplicate content, but URL's are tagged with campaign tags
Crawl diagnostics shows a lot of pages with duplicate content, but when I check the details, I see that it lists the same page but the url contains a campaign tag, so it's not really another page that is serving identical content... Is there a way to remove these pages out of the Crawl Diagnostics?
Moz Pro | | jorisbrabants0 -
Organic search visits in SEOMoz reporting
hi Are the organic search visit figures in SEOMoz traffic reporting unique or total visits ? cheers dan
Moz Pro | | Dan-Lawrence0 -
SEOmoz Campaign Setup: What would you do?
I've been using moz for nearly a year now. Initially when I set up our site (rac.com.au) it was as a single campaign for the rac.com.au domain. However, our business has a number of different product offerings and marketing teams (Insurance, finance, travel/tourism, automotive, security etc) - would I be better off setting each of these 'sub' brands as a campaign on their own using the 'folder' campaign setup? For example, have a campaign setup for the Insurance business (rac.com.au/insurance) folder, Finance business (rac.com.au/finance) and so forth? The advantage I see to the folder campaign setup is more specific reporting for each of the business units. Are there disadvantages to this? Any recommendations would be greatly appreciated. Cheers Ryan
Moz Pro | | Hutch_e0 -
Issue Using MozScape wtih SEOGadet's Link Excel Extension
Howdy Everyone! Since i've seen the MozCon presentation by Richard Baxter at SEOGadget, i've been completely amazed by the links extension for Excel. I tried this morning to give it a try. Unfortunately, I cannot get it working, and was hoping that someone here could help me out. I know it's out of the usual "realm" of questions, but I figure it's worth a try 🙂 I successfully installed the addon and entered my Access id (as "member-xxxxxxxxxx" member is in my id) and the secret key. I then downloaded the "OSE" excel spreadsheet" just to make sure I get all the calls right (as I know the doc works) Once I do this, and enter anything in, I get the "an error occured: the remote server returend an error: 401 unauthorized. I then went into the config file (as the setup doc suggests) and disabled run time caching (or at least set "SEOMOZ_API_use_cache_YN":"N") and the SEOMOZ_API_timeout to 100000 from 60000. I have also tried uninstalling and reinstalling the addin, along with regenerating the MozScape API Key Anyway, i'm not an excel wiz, and would appreciate any help that I could get on this. I'm also about to experiment with SEO Tools For Excel if anywone wants to check that out. Thanks in advance Zach Russell
Moz Pro | | Zachary_Russell0 -
Crawl Diagnostics Shows thousands of 302's from a single url. I'm confused
Hi guys I just ran my first campaign and the crawl diagnostics are showing some results I'm unfamiliar with.. In the warnings section it shows 2,838 redirects.. this is where I want to focus. When I click here it shows 5 redirects per page. When I go to click on page 2, or next page, or any other page than page 1 for that matter... this is where things get confusing. Nothing shows. Downloading the csv reveals that 2,834 of these are all showing: URL: http://www.mydomain.com/401/login.php url: http://www.mydomain.com/401/login.php referrer: http://www.mydomain.com/401/login.php location_header: http://www.mydomain.com/401/login.php I guess I'm just looking for an explanation as to why it's showing so many to the same page and what possible actions can be taken on my part to correct it (if needed). Thanks in advance
Moz Pro | | sethwb0 -
SEOmoz Dashboard Report: Crawl Diagnostic Summary
Hi there, I'm noticing that the total errors for our website has been going up and down drastically almost every other week. 4 weeks ago there were over 10,000 errors. 2 weeks ago there were barely 1,000 errors. Today I'm noticing it's back to over 12,000 errors. It says the majority of the errors are from duplicate page content & page title. We haven't made any changes to the titles or the content. Some insight and explanation for this would be much appreciated. Thanks, Gemma
Moz Pro | | RBA1 -
FB Shares in SEOmoz Keyword Analysis Tool
Hello everybody, I'm quite confused in how to understand "FB Shares" in SEOmoz Keyword Analysis Tool Report. As You may know, this section is under report's "Social Signals" category. Does this number counts whole FB shares of some website links by ALL FB users? Or is it a number of some website FB account posts? Some clearance would be very helpful.. The same question applies to other social signal reports: Tweets, Google+ shares. Thank You for your answer! Happy New Year!
Moz Pro | | jkundrotas0 -
Is there a whitelist of the RogerBot IP Addresses?
I'm all for letting Roger crawl my site, but it's not uncommon for malicious spiders to spoof the User-Agent string. Having a whitelist of Roger's IP addresses would be immensely useful!
Moz Pro | | EricCholis1
