Thinking aloud - what if WE could run rogerbot from our desktops?
-
Total, total noob question, I know - but is rogerbot performance bound because of bandwidth and processing capacity? I understand if it is, but I am wondering for those of us with very large sites if we would be able to offload the burden on SEOmoz resources by running our own local licensed version of rogerbot, crawl the sites we want and the upload the data to SEOmoz for analysis.
If this was possible would we be getting more immediate results?
-
On the topic of a private crawl (or distributed crawl), these are cool ideas, but not something we currently have in our plans. Having the crawl centralized allows us to store historic data and ensure polite crawling. This may take a little extra time (we are indeed doing a lot of crawls, as well as processing them and retrieving link data for each of them), but we are actively working on on our infrastructure to reduce our crawling and processing time.
While the first crawl does take a number of days, subsequent crawls are started on the same day each week, and should take roughly the same amount of time to complete, controlling for external factors. So in general you should have fresh crawl data right around weekly, give or take a day or two.
As for your specific crawls, I'd be happy to look into them for you. I'll send you a separate email to discuss.
-
Still waiting for my custom crawl launched on April 28th (it's May 2nd) to complete... seriously? If SEOmoz is overwhelmed, first of all, congratualations on being so popular, but I am not getting any timely data at all.
-
Or, instead of running Rogerbot locally, we could run some distributed processing (ie. BOINC) to help offload some of the pressure from your cloud?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When rogerbot tried to crawl my site it gets a 404\. Why?
When rogerbot tries to craw my site it tries http://website.com. My website then tries to redirect to http://www.website.com and is throwing a 404 and ends up not getting crawled. It also throws a 404 when trying to read my robots.txt file for some reason. We allow rogerbot user agent so unsure whats happening here. Is there something weird going on when trying to access my site without the 'www' that is causing the 404? Any insight is helpful here. Thanks,
Technical SEO | | BlakeBooth0 -
Fetch as Google Desktop Render Width?
What is Google's minimum desktop responsive webpage width? Fetch as Google for desktop is showing a skinnier version of our responsive page.
Technical SEO | | Desiree-CP0 -
What can be the cause for difference in local rankings between mobile and desktop?
I have a site that ranks differently for the same search term on mobile and desktop computer. I'm based in Glasgow, and the search term is (I've replaced the term with X's) XXXXXX XXXXX Glasgow Searching from a location in Glasgow: Desktop: Snackpack : 2, Organic : 6
Technical SEO | | johanisk
Mobile: Snackpack: 1, Organic : 10 I'm keen to improve on the Organic positions as this term is a lead generating one for me. My site is mobile friendly and scores 69/100 on the speed test. Do you think bumping the pagespeed well "into the green" would help improve it's position? Is there anything else I should look at?0 -
5 minutes riddle of a hacked website - 2nd run
Hi, i am just a common person and no developper. I am sure that one of you will no the solution to my problem. If you beleive that you are the one SEO expert i need, this is going to cost you 5 minutes of your time. Before i confuse you with my broken English and wrong terminology i have created a short video of my google stats. See attached youtube link. Intro: My site has been hacked, therefore i created a completely new site on a different server with fresh data. Unfortunately google does not seem to recognize all the changes. My website URL is: goo.gl/mJqXF4 _Sorry for posting in the wrong category. To answer the questions asked there:_My site was hacked via SQL Injects because it was outdated and without any security solution. My local files where not used in any way to create the new site with the exception of some NAS stored pictures. Support or educate me if you prefer, or just let me know what a complete cleanup and restoration of my #4 (keyword: proberaum frankfurt) and #2 (keyword: proberaum) ranking would cost me. Thank you in advance! E59V73q8jtQ
Technical SEO | | csabatoldi0 -
Can I "Run Macros" on my own?
I talked to the SEO company I am using and trying to get an understanding of what it is they are doing for me. They told me that one of the most important things they are doing is running macros. Is this something I could learn to do myself? What does it mean? How do I do it? How long does it take?? I have recently been educating myself on SEO and coded my website with metadata titles and descriptions. Is running macros something I can do on my own too? I guess I'd also just like to know what it is.
Technical SEO | | CapitolShine0 -
What do the mozzers think about this site of mine?
Hello SEO MOzzers, I am today wanting your feedback on a site that I recently went live with. My Google rankings for the main keywords are doing very well considering the site has been live for 3 weeks now. I of course have a list of items that i'm still working on, completing meta description tags, title tags, adding copy content to category pages, updating h1 tags, working on our backlinking campaign, etc. The site is www.profitness-supplies.com Let me know what you think Mozzers
Technical SEO | | seohive-2227200 -
Smart phone shows both mobile & desktop version of www.innoviafilms.com
Good morning from 2 degrees C mostly cloudy wetherby UK, Image this situation... Ive got two versions of a website: A desktop version & a mobile version, here are their respective urls: Desk top: http://www.innoviafilms.com Mobile: http://www.innoviafilms.com/m/home.aspx With blackberry bold in hand I type in via google search "Innovia films" i click on the search snippet and it renders the desktop version. But when i type the url directly in my Blackberry bold 9700 it returns the mobile version 😞 This confuses me. Having read this: http://googlewebmastercentral.blogspot.com/2011/02/making-websites-mobile- friendly.html I thought the desktop version would render in smartphones which the balckberry bold is. So my question is... "Why does the the mobile version only appear when you type in www.innoviafilms.com on a blackberry (and iphone 4) and the desktop version appears only when you enter "Innovia films" via google search" Any insights welcome 🙂
Technical SEO | | Nightwing0 -
Mobile site ranking in desktop searches
A robots.txt file is properly implemented on the mobile site to block Googlebot from crawling, yet, when running a site: query for the site in google, it still returns over 104,000 pages from the mobile site in the index. Why could this be happening?
Technical SEO | | craigsmith3330