Hi Greg

So our new index went up today. Looking at the last crawl date for www.easydigging.com we crawled the domain on 9/27 but did not crawl deep enough to find the original page containing the 301. So it may be a couple of more indexes before we crawl deeper to find it. What you would need to do help www.easydigging.com increase it's visibility/domain authority which will result in OSE finding more pages which will eventually find the link we need. The current page is no longer in our index as it not longer has any crawl authority and pretty soon it will not show any data in OSE unless it is re-crawled.

Once we can include "Just Discovered Links" into the main index, as Cyrus mentioned above, all you would need to do then is have any link tweeted from a high ranking account which will basically guarantee a crawl to that page.

Hi Greg

Would you be able to tell me when the 301 was added for http://www.easydigging.com/Garden_Cultivator/wheel_hoe_push_plow.html?

Hi Greg

We do set the expectations about how long it would take for us to process pages from our help hub:

Here's how we compile our index:

• We grab the most recent index.
• We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
• We start crawling from the top down until we've crawled 65,000,000,000 pages (which is about 25% the amount in Google's index).
• Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index.

We update our Mozscape Index every 4 weeks. Crawling the entire Internet to look for links takes 2-3 weeks, but our crawlers are always in motion. When we need to start processing, we grab all the data they have collected and start processing which can take up to 3 weeks to determine which of those links are the most important.

So it's very possible that it may be re-crawled with an index we are possibly updating today ahead of schedule. Like I mentioned earlier, it can take a while for metrics to transfer over as it isn't guaranteed that we will crawl the same page for each index update.

Hi Greg

It can take a while for all of your pages to be picked up in new indexes as I mentioned earlier. OSE will actually check for re-directs immediately but for the example page you provided, that page was last crawled on September 8th which did not have the re-direct at that time. It is not guaranteed that the same pages will be re-crawled during the next index updates which is why our metrics from seomoz.org moved to moz.com took a few months. OSE does not crawl pages in a hierarchal manner unlike our other crawler for diagnostics which follows links within the source code.

OSE needs a reason to re-crawl new and old pages so it can see the re-direct. We take the last 40 days of crawl data from the last Index if a page hasn't been re-crawled. We actually are about to release a new index today ahead of schedule so it's possible we may be able to re-crawl the old page. After I confirm a new index is complete I will re-check the old URL to see if OSE picked it up.

Hi Greg

Our last index update was on the 23rd and the next one is not scheduled until Nov 20th. Keep in mind that not all metrics will carry over, we will still index the site based on strong backlinks. For external links you would need to make sure the linking root domains linking the old pages update their links with the new ones as our crawler does not follow links and only sees what they are as is.

So what our index will eventually update/bring over:

Internal Followed Links
Total Internal Links

Updating the linking root domains with the new links will help with the external links.

Let me know if that makes sense!

Hi Greg

When setting up your re-direct not all metrics will transfer over to the new domain. It took us a few index updates to see most of our metrics to appear after we moved from seomoz.org to moz.com.

Would you be able to provide us with the URLs of the old site and the new site so we can make sure our crawler is picking up the 301s?

If you do not wish to share them here you can submit your information through our contact form:

http://moz.com/help/contact

Hi Trip/Monique!

Glad it is working on your end!

Hi Trip, no ETA yet as they are investigating. Once I have more info I will send an email

Hi Monique!

I have sent an email to you regarding your campaign not collecting data. Let me know if you have not received it and I will send another!

Cheers