Welcome to the Q&A Forum

Hey Mark!

Sorry you weren't satisfied with the answer - hopefully this large amount of detail will help [brace yourself]

First, a little history....

When we first built the web app, we had originally thought about emulating the tool Rank Tracker but in the campaign context.  Over 2 years (up to 2009) we had 85,000 keywords across all our customers, so we wanted to offer more functionality int he web app, so we allowed users to track up to 3 search engine for their keywords (this meant instead of just checking one rank, we generally check 3 ranks for every keyword).  We also increased the limit to many more keywords and allowed users to cut and past into the tool.

Since it was so easy to add keywords, and people watch Rand's video which made the suggestion to just "cut and paste all the words that drive traffic to your site", and pretty soon almost all our users were maxing out their keywords.

What we thought was a reasonable amount of rankings, quickly changed, and we were checking over 500,000-800,000 keywords every single night (and keep in mind this is just weekly checks).

The issues with lots of keywords

In order to do this politely (since we want to make sure we aren't abusing the search engines) we have to spread these out throughout the day, and for really heavy days collecting all those SERPs can take almost the entire day!

Since we don't want to prioritize certain users over others we have set the once a week limit which enables us to evenly distribute out collection over the whole week, and make sure that all rankings are always collected (since sometimes things can break when SERP layouts change, etc.)

Usability issues with different frequencies

Another challenge that has come up is that it is really important to a lot of users that all of their data updates on the same day.  This poses a problem when keywords and rankings are collected off cycle, since then graphs could end up with extra data points and the definition of "weekly change" throughout our UI becomes fuzzy (do you use the week delta, or do you use the last rank you checked?).  So we went for simple to make this more clear and usable for everyone.

So could we add more capacity, a new product, something?

Yes, to all of those questions.

We could make it so we could do a lot more keywords.  And we have considered on demand rank checking, or even just tracking keywords outside of the campaign.  However, it isn't something we are planning to do right now.

Instead we are focusing on improving the SERPs we do collect to include vertical results - like local, images and video - so you can track your performance in those areas.  We are also focusing on adding more functionality to the web app to make it even more valuable for each campaign - including adding social data, and more.

However, we definitely, definitely listen to our customers and use your feedback as a critical factor in all of our product planning efforts.  So if you think this is the wrong decision, or you want to tell us to build something - let us know here:

http://seomoz.zendesk.com/forums/293194-seomoz-pro-feature-requests

We couldn't build what we do without your input and it means a lot to us to hear your feedback.

Hope that helps!

Kate

Hi vishalkhialani!

I thought I would answer your question with some detail that might satisfy your curiosity (although I know more detailed blog posts are in the works).

For Linkscape:

At the heart of our architecture is our own column oriented data store - much like Vertica, although far more specialized for our use case - particularly in terms of the optimizations around compression and speed.

Each month we crawl between 1-2 petabytes of data, strip out the parts we care about (links, page attributes, etc) and then compute a link graph of how all those sites link to one another (typically between 40-90 billion urls) and then calculate our metrics using those results.  Once we have all of that we then precompute lots of views of the data, which is what gets displayed in Open SIte Explorer or retrieved via the Linkscape api.  These resulting views of the data is over 12 terabytes (and this is all raw text compressed data - so it is a LOT of information).  Making this fast and scalable is certainly a challenge.

For the crawling, we operate 10-20 boxes that crawl all the time.
For processing, we spin up between 40-60 instances to create the link graph, metrics and views.
And the API servers the index from S3 (Amazon's cloud storage) with 150-200 instances (but this was only 10 1 year ago, so we are seeing a lot of growth).

All of this is Linux and C++ (with some python thrown in here and there).

For custom crawl:

We use similar crawling algorithms to Linkscape, only we keep the  crawls per site, and also compute issues (like which pages are duplicates of one another).  Then each of those crawls are processed and precomputed to be served quickly and easily within the web app (so calculating the aggregates and deltas you see in the overview sections).

We use S3 for archival of all old crawls.  Cassandra for some of the details you see in detailed views, and a lot of the overviews and aggregates are served with the web app db.

Most of the code here is Ruby, except for the crawling and issue processing which is C++.  All of it runs on Linux.

Hope that helps explain!  Definitely let me know if you have more questions though!
Kate

Sean - I share Rand' sentiments, thanks so much for the suggestion!

We have considered distributed crawling in the past (or even distributed rank checking because then it would be in that user's locale) but there are a whole different set of challenges.  For example, you have to handle all the edge cases: what if a user's computer isn't on, or loses connectivity, what if we crawl too fast and the user gets blocked from a site, how do you write all that data securely?

Of course all of these concerns can be overcome, but right now we feel like we have a good handle on the problems, and it will be much faster for us to just fix what we have

Although, I know all of us are so appreciative of the ideas and support, and we will have something really great soon!

Hey Websensible,

No, we haven't fixed it - because it isn't really bug - we are just grouping all anchor text counts (internal and external) together at the moment.

I am guessing any similarities between the counts is just coincidence since we haven't changed this behavior.

-Kate

Hi Al,

I assume you mean more than one campaign in the pro web application?  If that is a correct assumption, the answer is yes - we think of campaigns as domains - so for each domain you have that will represent one campaign (and there are 5 offered in Pro - and here is a page with more info: http://www.seomoz.org/plans)

And you can always try it out by taking a free trial here: https://www.seomoz.org/cart/freetrial

Hope that helps, let me know if it didn't answer your question.

Kate

Hello Journeyman!

Great question - right now the anchor text contains both internal and external links and there isn't a good way to see which is which from the csv

We all here at SEOmoz recognize this shortcoming so we are planning to add the ability to drill down on the anchor text and see which links use which anchor text.  That feature is coming out sometime this summer

Hope that helps and have a great day!

Kate

Hey Mark!

Sorry you weren't satisfied with the answer - hopefully this large amount of detail will help [brace yourself]

First, a little history....

When we first built the web app, we had originally thought about emulating the tool Rank Tracker but in the campaign context.  Over 2 years (up to 2009) we had 85,000 keywords across all our customers, so we wanted to offer more functionality int he web app, so we allowed users to track up to 3 search engine for their keywords (this meant instead of just checking one rank, we generally check 3 ranks for every keyword).  We also increased the limit to many more keywords and allowed users to cut and past into the tool.

Since it was so easy to add keywords, and people watch Rand's video which made the suggestion to just "cut and paste all the words that drive traffic to your site", and pretty soon almost all our users were maxing out their keywords.

What we thought was a reasonable amount of rankings, quickly changed, and we were checking over 500,000-800,000 keywords every single night (and keep in mind this is just weekly checks).

The issues with lots of keywords

In order to do this politely (since we want to make sure we aren't abusing the search engines) we have to spread these out throughout the day, and for really heavy days collecting all those SERPs can take almost the entire day!

Since we don't want to prioritize certain users over others we have set the once a week limit which enables us to evenly distribute out collection over the whole week, and make sure that all rankings are always collected (since sometimes things can break when SERP layouts change, etc.)

Usability issues with different frequencies

Another challenge that has come up is that it is really important to a lot of users that all of their data updates on the same day.  This poses a problem when keywords and rankings are collected off cycle, since then graphs could end up with extra data points and the definition of "weekly change" throughout our UI becomes fuzzy (do you use the week delta, or do you use the last rank you checked?).  So we went for simple to make this more clear and usable for everyone.

So could we add more capacity, a new product, something?

Yes, to all of those questions.

We could make it so we could do a lot more keywords.  And we have considered on demand rank checking, or even just tracking keywords outside of the campaign.  However, it isn't something we are planning to do right now.

Instead we are focusing on improving the SERPs we do collect to include vertical results - like local, images and video - so you can track your performance in those areas.  We are also focusing on adding more functionality to the web app to make it even more valuable for each campaign - including adding social data, and more.

However, we definitely, definitely listen to our customers and use your feedback as a critical factor in all of our product planning efforts.  So if you think this is the wrong decision, or you want to tell us to build something - let us know here:

http://seomoz.zendesk.com/forums/293194-seomoz-pro-feature-requests

We couldn't build what we do without your input and it means a lot to us to hear your feedback.

Hope that helps!

Kate

Hi vishalkhialani!

I thought I would answer your question with some detail that might satisfy your curiosity (although I know more detailed blog posts are in the works).

For Linkscape:

At the heart of our architecture is our own column oriented data store - much like Vertica, although far more specialized for our use case - particularly in terms of the optimizations around compression and speed.

Each month we crawl between 1-2 petabytes of data, strip out the parts we care about (links, page attributes, etc) and then compute a link graph of how all those sites link to one another (typically between 40-90 billion urls) and then calculate our metrics using those results.  Once we have all of that we then precompute lots of views of the data, which is what gets displayed in Open SIte Explorer or retrieved via the Linkscape api.  These resulting views of the data is over 12 terabytes (and this is all raw text compressed data - so it is a LOT of information).  Making this fast and scalable is certainly a challenge.

For the crawling, we operate 10-20 boxes that crawl all the time.
For processing, we spin up between 40-60 instances to create the link graph, metrics and views.
And the API servers the index from S3 (Amazon's cloud storage) with 150-200 instances (but this was only 10 1 year ago, so we are seeing a lot of growth).

All of this is Linux and C++ (with some python thrown in here and there).

For custom crawl:

We use similar crawling algorithms to Linkscape, only we keep the  crawls per site, and also compute issues (like which pages are duplicates of one another).  Then each of those crawls are processed and precomputed to be served quickly and easily within the web app (so calculating the aggregates and deltas you see in the overview sections).

We use S3 for archival of all old crawls.  Cassandra for some of the details you see in detailed views, and a lot of the overviews and aggregates are served with the web app db.

Most of the code here is Ruby, except for the crawling and issue processing which is C++.  All of it runs on Linux.

Hope that helps explain!  Definitely let me know if you have more questions though!
Kate

Sean - I share Rand' sentiments, thanks so much for the suggestion!

We have considered distributed crawling in the past (or even distributed rank checking because then it would be in that user's locale) but there are a whole different set of challenges.  For example, you have to handle all the edge cases: what if a user's computer isn't on, or loses connectivity, what if we crawl too fast and the user gets blocked from a site, how do you write all that data securely?

Of course all of these concerns can be overcome, but right now we feel like we have a good handle on the problems, and it will be much faster for us to just fix what we have

Although, I know all of us are so appreciative of the ideas and support, and we will have something really great soon!