Thinking aloud - what if WE could run rogerbot from our desktops?
-
Total, total noob question, I know - but is rogerbot performance bound because of bandwidth and processing capacity? I understand if it is, but I am wondering for those of us with very large sites if we would be able to offload the burden on SEOmoz resources by running our own local licensed version of rogerbot, crawl the sites we want and the upload the data to SEOmoz for analysis.
If this was possible would we be getting more immediate results?
-
On the topic of a private crawl (or distributed crawl), these are cool ideas, but not something we currently have in our plans. Having the crawl centralized allows us to store historic data and ensure polite crawling. This may take a little extra time (we are indeed doing a lot of crawls, as well as processing them and retrieving link data for each of them), but we are actively working on on our infrastructure to reduce our crawling and processing time.
While the first crawl does take a number of days, subsequent crawls are started on the same day each week, and should take roughly the same amount of time to complete, controlling for external factors. So in general you should have fresh crawl data right around weekly, give or take a day or two.
As for your specific crawls, I'd be happy to look into them for you. I'll send you a separate email to discuss.
-
Still waiting for my custom crawl launched on April 28th (it's May 2nd) to complete... seriously? If SEOmoz is overwhelmed, first of all, congratualations on being so popular, but I am not getting any timely data at all.
-
Or, instead of running Rogerbot locally, we could run some distributed processing (ie. BOINC) to help offload some of the pressure from your cloud?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What can be the cause for difference in local rankings between mobile and desktop?
I have a site that ranks differently for the same search term on mobile and desktop computer. I'm based in Glasgow, and the search term is (I've replaced the term with X's) XXXXXX XXXXX Glasgow Searching from a location in Glasgow: Desktop: Snackpack : 2, Organic : 6
Technical SEO | | johanisk
Mobile: Snackpack: 1, Organic : 10 I'm keen to improve on the Organic positions as this term is a lead generating one for me. My site is mobile friendly and scores 69/100 on the speed test. Do you think bumping the pagespeed well "into the green" would help improve it's position? Is there anything else I should look at?0 -
What domain name do you think is better for SEO: sirocco-webdesign.com or sirocco-web-design.com?
Hello, I would appreciate it very much if you share with me your thoughts on what domain name I'd better pick out in terms of productive SEO: sirocco-webdesign.com or sirocco-web-design.com? I know hyphens are not good, but second domain looks better, I think.
Technical SEO | | kirupa0 -
Mobile Usability update mobile and desktop versions
Hello, A number of our clients have both m.website.com and website.com versions of their site. Per the latest update coming in April from Google, do I need to make sure all clients with just a desktop version are mobile-friendly (according to WT). Also, do I need to make both versions for those clients that have both m.website.com and website.com mobile-friendly or just the m.website.com? Thank you!!!!
Technical SEO | | lfrazer0 -
Do you think my client is being hit for duplicate content?
Wordpress website. The client's website is http://www.denenapoints.com/ The URL that we purchase so that we could setup the hosting account is http://houston-injury-lawyers.com, which shows 1 page indexed in Google when I search for site:http://houston-injury-lawyers.com On http://www.denenapoints.com/ there is <link rel="<a class="attribute-value">canonical</a>" href="http://houston-injury-lawyers.com/"> But on http://houston-injury-lawyers.com it says the same thing, <link rel="<a class="attribute-value">canonical</a>" href="http://houston-injury-lawyers.com/" /> Is this how it should be setup, assuming that we want everything to point to http://denenapoints.com/? Maybe we should do a 301 redirect to be 100% Sure? Hopefully I explained this well enough. Please let me know if anyone has any thoughts, thanks!
Technical SEO | | georgetsn0 -
Need advice on having customer stores running on my subdomain
We have an online store product and we're working on the SEO for our new domain (foo.com in this example.) Our customers have the ability to change the domain of their store but many of them will likely stick with the subdomains we give them (store1.foo.com) We could potentially have thousands of stores soon using our subdomain. Each of these stores will have a very small link at the bottom to our own domain but other than this, the content is completely user-generated and not under our control. Are there risks/problems associated with this type of strategy? If so, could we perhaps avoid them by using robots.txt to block entire site until they change to their own domain? TIA, Sean
Technical SEO | | schof0 -
Why is either Rogerbot or (if it is the case) Googlebots not recognizing keyword usage in my body text?
I have a client that does liposuction as one of their main services, they have been ranked in the top 1-5 for their keywords "sarasota liposuction" with different variations of the words for a long time, and suddenly have dropped about 10-12 places down to #15 in the engine. I went to investigate this and actually came to the "on-page analysis" tool for SEOmoz pro, where oddly enough it says that there is no mention of the target keyword in the body content (on-page analysis tool screenshot attached). I didn't quite understand why it would not recognize the obvious keywords in the body text so I went back to the page and inspected further. The keywords have an odd featured link that links up to an internally hosted keyword glossary for definitions of terms that people might not know directly. These definitions pop up in a lightbox upon clicking the keyword (liposuction lightbox screenshots attached). I have no idea why google would not recognize these words as they have the text in between the link, yet if there is something wrong with the code syntax etc. it might possibly hender the engine from seeing the body text of the link? any help would be greatly appreciated! Thank you so much! Phn2m Phn2m.png bWr5K.png V36CL.png
Technical SEO | | jbster130 -
Smart phone shows both mobile & desktop version of www.innoviafilms.com
Good morning from 2 degrees C mostly cloudy wetherby UK, Image this situation... Ive got two versions of a website: A desktop version & a mobile version, here are their respective urls: Desk top: http://www.innoviafilms.com Mobile: http://www.innoviafilms.com/m/home.aspx With blackberry bold in hand I type in via google search "Innovia films" i click on the search snippet and it renders the desktop version. But when i type the url directly in my Blackberry bold 9700 it returns the mobile version 😞 This confuses me. Having read this: http://googlewebmastercentral.blogspot.com/2011/02/making-websites-mobile- friendly.html I thought the desktop version would render in smartphones which the balckberry bold is. So my question is... "Why does the the mobile version only appear when you type in www.innoviafilms.com on a blackberry (and iphone 4) and the desktop version appears only when you enter "Innovia films" via google search" Any insights welcome 🙂
Technical SEO | | Nightwing0 -
Prevent mobile site from appearing in the sitelinks of desktop search
Hi, IWe have this mobile page that keeps on appearing in the google search. I even try to put it in the robots.txt to disallow the crawler but still it keeps on popping on the search results. How can I prevent it from displaying?
Technical SEO | | shebinhassan0