Thinking aloud - what if WE could run rogerbot from our desktops?
-
Total, total noob question, I know - but is rogerbot performance bound because of bandwidth and processing capacity? I understand if it is, but I am wondering for those of us with very large sites if we would be able to offload the burden on SEOmoz resources by running our own local licensed version of rogerbot, crawl the sites we want and the upload the data to SEOmoz for analysis.
If this was possible would we be getting more immediate results?
-
On the topic of a private crawl (or distributed crawl), these are cool ideas, but not something we currently have in our plans. Having the crawl centralized allows us to store historic data and ensure polite crawling. This may take a little extra time (we are indeed doing a lot of crawls, as well as processing them and retrieving link data for each of them), but we are actively working on on our infrastructure to reduce our crawling and processing time.
While the first crawl does take a number of days, subsequent crawls are started on the same day each week, and should take roughly the same amount of time to complete, controlling for external factors. So in general you should have fresh crawl data right around weekly, give or take a day or two.
As for your specific crawls, I'd be happy to look into them for you. I'll send you a separate email to discuss.
-
Still waiting for my custom crawl launched on April 28th (it's May 2nd) to complete... seriously? If SEOmoz is overwhelmed, first of all, congratualations on being so popular, but I am not getting any timely data at all.
-
Or, instead of running Rogerbot locally, we could run some distributed processing (ie. BOINC) to help offload some of the pressure from your cloud?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
For responsive site what should be lowest Screen Resolution for Desktop?
Hello Guys, Can you please share in details screen resolution I have to define for my responsive site for desktop, tablet & mobile. Your inputs are very valuable to me. Thanks! Micey
Technical SEO | | micey1230 -
How big of a problem is this? - link cannonicalization I think?
Hello! I'm new here. My moz Pro account is flagging my website for having 282 duplicate pages, which it is saying are critical issues. I looked at this and it seems like the problem is that many of my pages are being indexed like: www.joeborders.com and joeborders.com and oeborders.com/index. I think this is an issue with link cannonicalization...right? I contacted my website builder/host a while ago and they said they don't have a way to to link cannonicalization....is this a huge problem?...Is there some way to do it that I'm missing? Should i get a new website builder/host? 😛 for reference, this is what my builder/host said when I asked them about it: "Good question, at the moment we don't offer it, I will add it to our feature request list, as I think it would be a good idea. In a traditional hosting environment this would be using a htaccess file, since we are in ruby on rails environment we would need come up with a custom solution."
Technical SEO | | joebordersmft0 -
What domain name do you think is better for SEO: sirocco-webdesign.com or sirocco-web-design.com?
Hello, I would appreciate it very much if you share with me your thoughts on what domain name I'd better pick out in terms of productive SEO: sirocco-webdesign.com or sirocco-web-design.com? I know hyphens are not good, but second domain looks better, I think.
Technical SEO | | kirupa0 -
5 minutes riddle of a hacked website - 2nd run
Hi, i am just a common person and no developper. I am sure that one of you will no the solution to my problem. If you beleive that you are the one SEO expert i need, this is going to cost you 5 minutes of your time. Before i confuse you with my broken English and wrong terminology i have created a short video of my google stats. See attached youtube link. Intro: My site has been hacked, therefore i created a completely new site on a different server with fresh data. Unfortunately google does not seem to recognize all the changes. My website URL is: goo.gl/mJqXF4 _Sorry for posting in the wrong category. To answer the questions asked there:_My site was hacked via SQL Injects because it was outdated and without any security solution. My local files where not used in any way to create the new site with the exception of some NAS stored pictures. Support or educate me if you prefer, or just let me know what a complete cleanup and restoration of my #4 (keyword: proberaum frankfurt) and #2 (keyword: proberaum) ranking would cost me. Thank you in advance! E59V73q8jtQ
Technical SEO | | csabatoldi0 -
Does Google think My main keyword is my Company name due to over use in anchor text?
OK, i have recently noticed google is adding a keyword to my titles... I have attached an image which shows the results from a google site:mywebsite search... 8 out of the 10 results on the screen shot clearly have the keyword ( - CCTV Cameras) added to the end of the title. Now a lot of those pages are support pages i.e "Help & Support", "returns" etc etc.. nothing really to do with cctv cameras. The only thing i can think of, is the most common anchor text used on backlinks to my website (according to siteexplorer) is the keyword cctv cameras... (i am currently working on removing a load of the backlinks to my site due to what i think is some kind of google penalty as the anchor text is all keywords etc...) But it seems strange if google has started to discount these backlinks, that it then uses that keyword to add to my titles - unless of course, google thinks that it is my brand name... Any body have any thoughts on this? FqwooQY.gif
Technical SEO | | isntworkdull1 -
Need advice on having customer stores running on my subdomain
We have an online store product and we're working on the SEO for our new domain (foo.com in this example.) Our customers have the ability to change the domain of their store but many of them will likely stick with the subdomains we give them (store1.foo.com) We could potentially have thousands of stores soon using our subdomain. Each of these stores will have a very small link at the bottom to our own domain but other than this, the content is completely user-generated and not under our control. Are there risks/problems associated with this type of strategy? If so, could we perhaps avoid them by using robots.txt to block entire site until they change to their own domain? TIA, Sean
Technical SEO | | schof0 -
Prevent mobile site from appearing in the sitelinks of desktop search
Hi, IWe have this mobile page that keeps on appearing in the google search. I even try to put it in the robots.txt to disallow the crawler but still it keeps on popping on the search results. How can I prevent it from displaying?
Technical SEO | | shebinhassan0 -
.us domains vs .com - What does Google Think?
Suppose I had 2 domains, carloans.us & carloans.com with exactly the same links profiles, and content (not duplicate but you know what I mean). Would Google favour the .com domain? In my experience, yes. But I might be wrong?
Technical SEO | | Tom-R
Same with other not so standard domains like .biz etc. Am I right to believe that Google can prefer the more common domain extensions?0