Thinking aloud - what if WE could run rogerbot from our desktops?
-
Total, total noob question, I know - but is rogerbot performance bound because of bandwidth and processing capacity? I understand if it is, but I am wondering for those of us with very large sites if we would be able to offload the burden on SEOmoz resources by running our own local licensed version of rogerbot, crawl the sites we want and the upload the data to SEOmoz for analysis.
If this was possible would we be getting more immediate results?
-
On the topic of a private crawl (or distributed crawl), these are cool ideas, but not something we currently have in our plans. Having the crawl centralized allows us to store historic data and ensure polite crawling. This may take a little extra time (we are indeed doing a lot of crawls, as well as processing them and retrieving link data for each of them), but we are actively working on on our infrastructure to reduce our crawling and processing time.
While the first crawl does take a number of days, subsequent crawls are started on the same day each week, and should take roughly the same amount of time to complete, controlling for external factors. So in general you should have fresh crawl data right around weekly, give or take a day or two.
As for your specific crawls, I'd be happy to look into them for you. I'll send you a separate email to discuss.
-
Still waiting for my custom crawl launched on April 28th (it's May 2nd) to complete... seriously? If SEOmoz is overwhelmed, first of all, congratualations on being so popular, but I am not getting any timely data at all.
-
Or, instead of running Rogerbot locally, we could run some distributed processing (ie. BOINC) to help offload some of the pressure from your cloud?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can slow mobile page speed affect desktop search results?
I heard recently from an SEO friend that with Google's recent update, mobile page speed now affects desktop results. Our site is relatively slow on mobile, and I wanted to check! Thank you!
Technical SEO | | lauraballer1 -
How big of a problem is this? - link cannonicalization I think?
Hello! I'm new here. My moz Pro account is flagging my website for having 282 duplicate pages, which it is saying are critical issues. I looked at this and it seems like the problem is that many of my pages are being indexed like: www.joeborders.com and joeborders.com and oeborders.com/index. I think this is an issue with link cannonicalization...right? I contacted my website builder/host a while ago and they said they don't have a way to to link cannonicalization....is this a huge problem?...Is there some way to do it that I'm missing? Should i get a new website builder/host? 😛 for reference, this is what my builder/host said when I asked them about it: "Good question, at the moment we don't offer it, I will add it to our feature request list, as I think it would be a good idea. In a traditional hosting environment this would be using a htaccess file, since we are in ruby on rails environment we would need come up with a custom solution."
Technical SEO | | joebordersmft0 -
Do you think have to re-submit my site to search engines after I made improvements?
Some time ago I started to do SEO for a one-page website and didn't get any positive result: no traffic, no filled in online booking form (yet another, multiple page website offering the same service yielded in multiple filled-in "schedule an appointment" forms). I found out my one-page website was considered to be "keyword-spamming" and converted it to a multiple page one. Its domain authority went up, but it doesn't still bring any traffic. I am thinking maybe I have to let the search engines know that it has been updated so they stop penalizing it? Do you think it might help and if yes, what exactly I should do? Will be thankful very much for any suggestion!
Technical SEO | | kirupa0 -
We have 2 versions of URLs. we have the mobile and the desktop. is that a duplicate content?
Hi, Our website has two version of URLs. dektop: www.myexample.com and mobile www.myexample.com/m If you go to our site from a mobile device you will land on our mobile URL, if you go to our site from desktop computer you will land on a regular URL. Both urls have the same content. Is that considered duplicate? If yes, then what can I do to fix it? Also, both URLs are indexed by google. We have two separate XML sitemaps- one for desktop and one for mobile. Is that a good SEO practice?
Technical SEO | | Armen-SEO0 -
Do you think this site has been hit by penguin?
Hi Guys, I need some opinion on a website i am working on www.colourbnners.co.uk They updated their website in August but the company they used did not take into account the URL structure and hence there's a massive loss in links in August time. They also dropped off Google for all their key terms except their brand name 'colour banners'
Technical SEO | | gezzagregz
Since then, they have implemented a 301 redirect. Some key points They have not received any manual warnings in WMT I have disavowed some poor quality links that they have built over the years I am building high quality links quite selectively/slowly There were a lot of duplicate content issues - these have been resolved now.
So my question to you SEO pros is do you think its penguin? or something that i am missing?
If it is penguin, what is the best form of attack to get it removed? regards gezzagrez0 -
Mobile site ranking instead of/as well as desktop site in desktop SERPS
I have just noticed that the mobile version of my site is sometimes ranking in the desktop serps either instead of as well as the desktop site. It is not something that I have noticed in the past as it doesn't happen with the keywords that I track, which are highly competitive. It is happening for results that include our brand name, e.g '[brand name][search term]'. The mobile site is served with mobile optimised content from another URL. e.g wwww.domain.com/productpage redirects to m.domain.com/productpage for mobile. Sometimes I am only seen the mobile URL in the desktop SERPS, other times I am seeing both the desktop and mobile URL for the same product. My understanding is that the mobile URL should not be ranking at all in desktop SERPS, could we be being penalised for either bad redirects or duplicate content? Any ideas as to how I could further diagnose and solve the problem if you do believe that it could be harming rankings?
Technical SEO | | pugh0 -
Want to Target Mobile site for Google Mobile Version and Desktop Site for Google Desktop Version
I have ecommerce site with both mobile version and desktop version. Mobile version starts with m.example.com and full version starts with www.example.com I am using same content through out both site and using 301 redirection by detecting user agent vice-versa. My both sites are accessible to crawl by any google spider. I have submitted both sites's sitemap to GWT and mobile site having mobile sitemap xml, so google can easily recognize my mobile site. Is it going to help to rank my both sites as per my expectation? I need to rank for mobile site in Google mobile and ranking for desktop site in Google desktop version. Some of pages of my mobile site are started to appearing in Google desktop version. So how I can stop them to appear in Google desktop? Your comments are highly welcome.
Technical SEO | | Hexpress0 -
Does running 2 domains from the same website harm my seo efforts?
Hi I run a website which has 2 domains running from it. One domain has a .co.uk extension and shows products aimed at the UK market. The other domain is a .com which shows products aimed at the US market. As both domains run from the same website, the content is mostly identical APART from the product listings as they change depending upon whether you're in the UK or US. My reason for doing is this mainly for paid search purposes as I can have a more accurate domain depending upon which country I target. Both sites are set to encourage the bots to index it, and in google WMT each domain is targeted to its specific country. So, my questions are, Is this set-up possibly damaging my SEO efforts for either or both websites? If so, would setting one domain to no-index, improve SEO efforts for the other? Thanks in advance for any replies! Cheers Joe
Technical SEO | | joeprice0