301 Redirects: How long for Google to recognize? How long for Moz/OSE to recognize?

[GregB123]

I redirected a few pages that had funky URL issues (capitals and underscores and useless words) to the same page renamed with proper URLs. The sitemaps were changed also, and re-submitted.

Like this:  mydomain.com/Not_Pertinent_Words.html  >>>  mydomain.com/good-words.html

Google seems to have found them and changed the search engine results listing in about 8 days.

But it's been about a month now and Moz and OSE still have not transferred all the strength and link data from the old URLs to the new ones.

Question 1: How long does it usually take Google to transfer all the link and strength data for a 301?

Question 2: How long does it take Moz and OSE to do the same? Is there something I need to do to tell them about the changes?

[SEOGoddess]

I had a similar issue with usedcars.com. We had several hundred thousand URLs that were redirecting from parameter driven URLs, to clean hierarchical URLs with a huge initiative to clean them all up in a few months. It took well over a year and still didn't see them getting recognized.

I blogged the process - but the part about what I did to speed it up:

---

I grabbed one of my many impulsive purchased domains and quickly set up hosting and an old school html site that consisted of one page. I then exported all of the links on the Google "site:" search through a Firefox plugin called SEOquake that exports the results into a csv file. It's not the prettiest, and there was a lot of work still needed to get to just the URLs, but it was the best solution I could find (note: if any SEO reading this knows of an easier way to do this - please add to the comments for prosperity). I then parsed out the parameters in the URLs in a separate document and used those as the anchor text for each URL. Finally, using excel I then concatenated the URLs and parameters (that were now anchor text) into an html href string.
Then copying and pasting the "string" column into the html code, the page looked like:
The page wasn't the prettiest, and it had thousands of links (the above is just an example) so it was bad all around, but the point was to get those links crawled by Google.

Of course every SEO knows that you can't just build a website and expect it to immediately get crawled - right?

So I set it up in Google Webmaster Tools and submitted the page to the the index:
I even got more fancy to ensure Google would see the page and crawl all of those old URLs and +1'd it on Google.

Did it work?

I checked the URLs this evening to see how many Google is seeing and the number has dropped from 550,000 to now only 175.

I took the domain off of the server, and now have it parked elsewhere (back where it belongs) and removed the webmaster tools account. All traces of it ever existing are now gone, and the small moment of my attempt to get those URLs removed has passed.

---

It's very black hat - and could potentially get you in trouble if not approached carefully. But - it worked...

[Cyrus-Shepard]

Hi Gregory,

Great suggestion. It might be a little challenging to mix the private crawl (Your campaign) with the public one (OSE) but it's certainly possible to cross-pollinate data.

In the meantime, we're working on better incorporating our "just discovered links" in OSE to be included in the main index. This means that anytime a link is tweeted (our source of discovery for Just Discovered Links) it will be included for crawling in the general index.

Regardless, thanks for your input into this matter, and I apologize for it being a frustrating experience.

Cheers,

Cyrus

[DavidLee]

Hi Greg

So our new index went up today. Looking at the last crawl date for www.easydigging.com we crawled the domain on 9/27 but did not crawl deep enough to find the original page containing the 301. So it may be a couple of more indexes before we crawl deeper to find it. What you would need to do help www.easydigging.com increase it's visibility/domain authority which will result in OSE finding more pages which will eventually find the link we need. The current page is no longer in our index as it not longer has any crawl authority and pretty soon it will not show any data in OSE unless it is re-crawled.

Once we can include "Just Discovered Links" into the main index, as Cyrus mentioned above, all you would need to do then is have any link tweeted from a high ranking account which will basically guarantee a crawl to that page.

[GregB123]

I don't have documentation, but the new page it redirects to has this tag:

So I would guess I did the 301 redirect the to this new page at the same time I launched the new page (there was no major design changes - just shortening the URL and getting rid of the capitals and underscores)

[DavidLee]

Hi Greg

Would you be able to tell me when the 301 was added for http://www.easydigging.com/Garden_Cultivator/wheel_hoe_push_plow.html?

[DavidLee]

Hi Greg

We do set the expectations about how long it would take for us to process pages from our help hub:

Here's how we compile our index:

• We grab the most recent index.
• We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
• We start crawling from the top down until we've crawled 65,000,000,000 pages (which is about 25% the amount in Google's index).
• Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index.

We update our Mozscape Index every 4 weeks. Crawling the entire Internet to look for links takes 2-3 weeks, but our crawlers are always in motion. When we need to start processing, we grab all the data they have collected and start processing which can take up to 3 weeks to determine which of those links are the most important.

So it's very possible that it may be re-crawled with an index we are possibly updating today ahead of schedule. Like I mentioned earlier, it can take a while for metrics to transfer over as it isn't guaranteed that we will crawl the same page for each index update.

[GregB123]

Thanks David. That does explain the issue better.

However, the occasional really long lag time for OSE to re-crawled pages that have been 301 redirected is still a problem for those of use who subscribe to Moz and are trying to use it to monitor and improve our SEO. But I think there is an easy solution...

Give Moz subscribers a new tool like GWT's "Fetch as Google" - but call it "Fetch as Moz" and have it trigger BOTH the Moz spider and the OSE spider to go re-crawl that page. This would let us quickly and positively know that our data for 301'd pages has been transferred to the new page.

Can this be done?

[DavidLee]

Hi Greg

It can take a while for all of your pages to be picked up in new indexes as I mentioned earlier. OSE will actually check for re-directs immediately but for the example page you provided, that page was last crawled on September 8th which did not have the re-direct at that time. It is not guaranteed that the same pages will be re-crawled during the next index updates which is why our metrics from seomoz.org moved to moz.com took a few months. OSE does not crawl pages in a hierarchal manner unlike our other crawler for diagnostics which follows links within the source code.

OSE needs a reason to re-crawl new and old pages so it can see the re-direct. We take the last 40 days of crawl data from the last Index if a page hasn't been re-crawled. We actually are about to release a new index today ahead of schedule so it's possible we may be able to re-crawl the old page. After I confirm a new index is complete I will re-check the old URL to see if OSE picked it up.

[GregB123]

Hi David,

Sorry to be blunt here, but no it doesn't make sense - and I also think it is incorrect...

Websites do internal 301 redirects all the time (redirecting from old page to new page within the same site) BUT your explanation above appears to say that OSE does not follow those redirects and move the data from old to new url ?  That would make for a pretty useless SEO program...

Your message also appears to say that if we have to go contact everybody that links to www.mysite.com/old-page.html and have them change their link to www.mysite.com/new-page.html  ? That's just silly...

I think your explanation is incorrect because SOME of the 301 re-directs I have done on my site have passed the data perfectly in OSE and Moz. Run OSE on these two URLs and you will see:  Old URL = www.bestdryingrack.com/clothes_drying_rack.html    New URL it was 301'ed to = www.bestdryingrack.com/clothes-drying-rack-main.html

So it appears that inconsistency is the a big problem here. That should be some firm rule like: "Moz will transfer all data across 301 redirects on the same domain within 1 month"  This would let us analyze or sites without trying to keep track of silly situations of which new pages have the proper data and which ones we still need to fudge between old and new page data.

I hope this all makes sense...

[DavidLee]

Hi Greg

Our last index update was on the 23rd and the next one is not scheduled until Nov 20th. Keep in mind that not all metrics will carry over, we will still index the site based on strong backlinks. For external links you would need to make sure the linking root domains linking the old pages update their links with the new ones as our crawler does not follow links and only sees what they are as is.

So what our index will eventually update/bring over:

Internal Followed Links
Total Internal Links

Updating the linking root domains with the new links will help with the external links.

Let me know if that makes sense!

[GregB123]

Hi David,

I waited another week and just now checked the new and redirected URL again. The data STILL has not transferred over to the new URL.

It's been a long time. This excessive lag time in moving data across re-directs really needs to be fixed.

Personally I can mentally keep track of it, but what about the other SEO programs out there that are using Moz data? They have to experiencing "garbage in - garbage out" problems on any analysis involving re-directed URL's.

Can you check into this and let us know what's up?

[GregB123]

Hello David, Thanks for jumping in!

The old URL was  http://www.easydigging.com/Garden_Cultivator/wheel_hoe_push_plow.html  and the new one it 301's to is  http://www.easydigging.com/wheel-hoe-hoss.html   Please reply back here so I know what the situation is.

Since it appears that Index updates only happen monthly, and it takes a few months for data to fully migrate over to redirected URLs, it would be really nice to change the little message that shows on OSE when an analyzed page is redirected (currently something like "Do you want to see the data for page this redirects to") to something that describes the lag, like.. "It takes a few months for Moz data to transfer to redirected URLs, so be sure to check both the old and the new URL since the one with higher statistics is the accurate one"

[DavidLee]

Hi Greg

When setting up your re-direct not all metrics will transfer over to the new domain. It took us a few index updates to see most of our metrics to appear after we moved from seomoz.org to moz.com.

Would you be able to provide us with the URLs of the old site and the new site so we can make sure our crawler is picking up the 301s?

If you do not wish to share them here you can submit your information through our contact form:

http://moz.com/help/contact

[GregB123]

Hi again Chris.  Mostly I was wanting to use Moz to evaluate my site for SEO reasons and frustrated that I kept having to use "old" page URLs to get data.

Also I have just started to experiment with the Penguin Analysis tool from Virante, and it uses Moz data mixed with data from other SEO programs - so I get corrupt results there whether I input the old URL or the new URL.  http://www.penguinanalysis.com/

If they really updated the Mozscape index on the 18th, it seems it should show by now...

[Chris_CM]

I wouldn't worry about the Mozscape index not showing your changes.  I put up a client site on the 8th and, despite thousands of visitors and some very strong backlinks, there is no data for it on the OSE.  Like I said, try Majestic, their tool does show this recent client site and the link data is accurate, so it may display more accurate citation and trust flow (Majestic authority metrics) for your site after the redirect than OSE.

[GregB123]

Thanks Chris. But unfortunately that makes the situation even worse. I did these 301 redirects weeks ago, and your quote say that there was a Mozscape Index update was a few days ago on October 18th.

I just checked again in OSE and the old page data and strength still has not transferred over to the new page. I have attached a screenshot from OSE showing this.

Any ideas?

page-difference.jpg

[Chris_CM]

Question 1:  If Google has correctly updated the index, I would imagine that it has redirected the link profile as well.

Question 2:

"Just-Discovered Links index updated continuously. Last Mozscape index update: October 18, 2013. Next Mozscape index update: November 15, 2013"  From OSE.

If you're looking for results on a more recent site, try Majestic's Site Explorer.  It is in my experience more accurate with newer sites and fresh redirects.

Hope this helps.

Chris