301 Redirects: How long for Google to recognize? How long for Moz/OSE to recognize?
-
I redirected a few pages that had funky URL issues (capitals and underscores and useless words) to the same page renamed with proper URLs. The sitemaps were changed also, and re-submitted.
Like this: mydomain.com/Not_Pertinent_Words.html >>> mydomain.com/good-words.html
Google seems to have found them and changed the search engine results listing in about 8 days.
But it's been about a month now and Moz and OSE still have not transferred all the strength and link data from the old URLs to the new ones.
Question 1: How long does it usually take Google to transfer all the link and strength data for a 301?
Question 2: How long does it take Moz and OSE to do the same? Is there something I need to do to tell them about the changes?
-
I had a similar issue with usedcars.com. We had several hundred thousand URLs that were redirecting from parameter driven URLs, to clean hierarchical URLs with a huge initiative to clean them all up in a few months. It took well over a year and still didn't see them getting recognized.
I blogged the process - but the part about what I did to speed it up:
I grabbed one of my many impulsive purchased domains and quickly set up hosting and an old school html site that consisted of one page. I then exported all of the links on the Google "site:" search through a Firefox plugin called SEOquake that exports the results into a csv file. It's not the prettiest, and there was a lot of work still needed to get to just the URLs, but it was the best solution I could find (note: if any SEO reading this knows of an easier way to do this - please add to the comments for prosperity). I then parsed out the parameters in the URLs in a separate document and used those as the anchor text for each URL. Finally, using excel I then concatenated the URLs and parameters (that were now anchor text) into an html href string.
Then copying and pasting the "string" column into the html code, the page looked like:
The page wasn't the prettiest, and it had thousands of links (the above is just an example) so it was bad all around, but the point was to get those links crawled by Google.Of course every SEO knows that you can't just build a website and expect it to immediately get crawled - right?
So I set it up in Google Webmaster Tools and submitted the page to the the index:
I even got more fancy to ensure Google would see the page and crawl all of those old URLs and +1'd it on Google.Did it work?
I checked the URLs this evening to see how many Google is seeing and the number has dropped from 550,000 to now only 175.
I took the domain off of the server, and now have it parked elsewhere (back where it belongs) and removed the webmaster tools account. All traces of it ever existing are now gone, and the small moment of my attempt to get those URLs removed has passed.
It's very black hat - and could potentially get you in trouble if not approached carefully. But - it worked...
-
Hi Gregory,
Great suggestion. It might be a little challenging to mix the private crawl (Your campaign) with the public one (OSE) but it's certainly possible to cross-pollinate data.
In the meantime, we're working on better incorporating our "just discovered links" in OSE to be included in the main index. This means that anytime a link is tweeted (our source of discovery for Just Discovered Links) it will be included for crawling in the general index.
Regardless, thanks for your input into this matter, and I apologize for it being a frustrating experience.
Cheers,
Cyrus
-
Hi Greg
So our new index went up today. Looking at the last crawl date for www.easydigging.com we crawled the domain on 9/27 but did not crawl deep enough to find the original page containing the 301. So it may be a couple of more indexes before we crawl deeper to find it. What you would need to do help www.easydigging.com increase it's visibility/domain authority which will result in OSE finding more pages which will eventually find the link we need. The current page is no longer in our index as it not longer has any crawl authority and pretty soon it will not show any data in OSE unless it is re-crawled.
Once we can include "Just Discovered Links" into the main index, as Cyrus mentioned above, all you would need to do then is have any link tweeted from a high ranking account which will basically guarantee a crawl to that page.
-
I don't have documentation, but the new page it redirects to has this tag:
So I would guess I did the 301 redirect the to this new page at the same time I launched the new page (there was no major design changes - just shortening the URL and getting rid of the capitals and underscores)
-
Hi Greg
Would you be able to tell me when the 301 was added for http://www.easydigging.com/Garden_Cultivator/wheel_hoe_push_plow.html?
-
Hi Greg
We do set the expectations about how long it would take for us to process pages from our help hub:
Here's how we compile our index:
- We grab the most recent index.
- We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
- We start crawling from the top down until we've crawled 65,000,000,000 pages (which is about 25% the amount in Google's index).
- Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index.
We update our Mozscape Index every 4 weeks. Crawling the entire Internet to look for links takes 2-3 weeks, but our crawlers are always in motion. When we need to start processing, we grab all the data they have collected and start processing which can take up to 3 weeks to determine which of those links are the most important.
So it's very possible that it may be re-crawled with an index we are possibly updating today ahead of schedule. Like I mentioned earlier, it can take a while for metrics to transfer over as it isn't guaranteed that we will crawl the same page for each index update.
-
Thanks David. That does explain the issue better.
However, the occasional really long lag time for OSE to re-crawled pages that have been 301 redirected is still a problem for those of use who subscribe to Moz and are trying to use it to monitor and improve our SEO. But I think there is an easy solution...
Give Moz subscribers a new tool like GWT's "Fetch as Google" - but call it "Fetch as Moz" and have it trigger BOTH the Moz spider and the OSE spider to go re-crawl that page. This would let us quickly and positively know that our data for 301'd pages has been transferred to the new page.
Can this be done?
-
Hi Greg
It can take a while for all of your pages to be picked up in new indexes as I mentioned earlier. OSE will actually check for re-directs immediately but for the example page you provided, that page was last crawled on September 8th which did not have the re-direct at that time. It is not guaranteed that the same pages will be re-crawled during the next index updates which is why our metrics from seomoz.org moved to moz.com took a few months. OSE does not crawl pages in a hierarchal manner unlike our other crawler for diagnostics which follows links within the source code.
OSE needs a reason to re-crawl new and old pages so it can see the re-direct. We take the last 40 days of crawl data from the last Index if a page hasn't been re-crawled. We actually are about to release a new index today ahead of schedule so it's possible we may be able to re-crawl the old page. After I confirm a new index is complete I will re-check the old URL to see if OSE picked it up.
-
Hi David,
Sorry to be blunt here, but no it doesn't make sense - and I also think it is incorrect...
Websites do internal 301 redirects all the time (redirecting from old page to new page within the same site) BUT your explanation above appears to say that OSE does not follow those redirects and move the data from old to new url ? That would make for a pretty useless SEO program...
Your message also appears to say that if we have to go contact everybody that links to www.mysite.com/old-page.html and have them change their link to www.mysite.com/new-page.html ? That's just silly...
I think your explanation is incorrect because SOME of the 301 re-directs I have done on my site have passed the data perfectly in OSE and Moz. Run OSE on these two URLs and you will see: Old URL = www.bestdryingrack.com/clothes_drying_rack.html New URL it was 301'ed to = www.bestdryingrack.com/clothes-drying-rack-main.html
So it appears that inconsistency is the a big problem here. That should be some firm rule like: "Moz will transfer all data across 301 redirects on the same domain within 1 month" This would let us analyze or sites without trying to keep track of silly situations of which new pages have the proper data and which ones we still need to fudge between old and new page data.
I hope this all makes sense...
-
Hi Greg
Our last index update was on the 23rd and the next one is not scheduled until Nov 20th. Keep in mind that not all metrics will carry over, we will still index the site based on strong backlinks. For external links you would need to make sure the linking root domains linking the old pages update their links with the new ones as our crawler does not follow links and only sees what they are as is.
So what our index will eventually update/bring over:
Internal Followed Links
Total Internal LinksUpdating the linking root domains with the new links will help with the external links.
Let me know if that makes sense!
-
Hi David,
I waited another week and just now checked the new and redirected URL again. The data STILL has not transferred over to the new URL.
It's been a long time. This excessive lag time in moving data across re-directs really needs to be fixed.
Personally I can mentally keep track of it, but what about the other SEO programs out there that are using Moz data? They have to experiencing "garbage in - garbage out" problems on any analysis involving re-directed URL's.
Can you check into this and let us know what's up?
-
Hello David, Thanks for jumping in!
The old URL was http://www.easydigging.com/Garden_Cultivator/wheel_hoe_push_plow.html and the new one it 301's to is http://www.easydigging.com/wheel-hoe-hoss.html Please reply back here so I know what the situation is.
Since it appears that Index updates only happen monthly, and it takes a few months for data to fully migrate over to redirected URLs, it would be really nice to change the little message that shows on OSE when an analyzed page is redirected (currently something like "Do you want to see the data for page this redirects to") to something that describes the lag, like.. "It takes a few months for Moz data to transfer to redirected URLs, so be sure to check both the old and the new URL since the one with higher statistics is the accurate one"
-
Hi Greg
When setting up your re-direct not all metrics will transfer over to the new domain. It took us a few index updates to see most of our metrics to appear after we moved from seomoz.org to moz.com.
Would you be able to provide us with the URLs of the old site and the new site so we can make sure our crawler is picking up the 301s?
If you do not wish to share them here you can submit your information through our contact form:
-
Hi again Chris. Mostly I was wanting to use Moz to evaluate my site for SEO reasons and frustrated that I kept having to use "old" page URLs to get data.
Also I have just started to experiment with the Penguin Analysis tool from Virante, and it uses Moz data mixed with data from other SEO programs - so I get corrupt results there whether I input the old URL or the new URL. http://www.penguinanalysis.com/
If they really updated the Mozscape index on the 18th, it seems it should show by now...
-
I wouldn't worry about the Mozscape index not showing your changes. I put up a client site on the 8th and, despite thousands of visitors and some very strong backlinks, there is no data for it on the OSE. Like I said, try Majestic, their tool does show this recent client site and the link data is accurate, so it may display more accurate citation and trust flow (Majestic authority metrics) for your site after the redirect than OSE.
-
Thanks Chris. But unfortunately that makes the situation even worse. I did these 301 redirects weeks ago, and your quote say that there was a Mozscape Index update was a few days ago on October 18th.
I just checked again in OSE and the old page data and strength still has not transferred over to the new page. I have attached a screenshot from OSE showing this.
Any ideas?
-
Question 1: If Google has correctly updated the index, I would imagine that it has redirected the link profile as well.
Question 2:
"Just-Discovered Links index updated continuously. Last Mozscape index update: October 18, 2013. Next Mozscape index update: November 15, 2013" From OSE.
If you're looking for results on a more recent site, try Majestic's Site Explorer. It is in my experience more accurate with newer sites and fresh redirects.
Hope this helps.
Chris
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page Authority higher on my old (redirected) domain than new domain?
Hi everyone. I moved the domain my blog was on about 18 months ago, and it's ranking in Google never recovered. I've noticed in the 'Inbound Links' tool that my old domain (jamescrowley.co.uk) shows a page authority of 26, while my new domain (jamescrowley.net) shows a page authority of 15. Any idea why that would be? 301 redirects have been in place for the whole 18 months, so I'd have thought the authority would have been 'passed on' by now? Many thanks James
Link Explorer | | james.crowley0 -
Can I change the Link Explorer from check with Google US?
I'm trying to check the ranking keywords for a specific article with the Link Explorer, but when I hover over Ranking Keywords on the Overview it tells me that it's looking on Google US. We're a UK based company and the keywords we're trying to rank for are UK based - is there a way of changing this to match?
Link Explorer | | Kestra0 -
Moz crawling doesn't show all of my pages
Hello, I'm trying to make an SEO on-pages audit of my website. When using the SEO crawl, i see only one page while i have much more pages in this website (200+). Is it an issue with the sitemap ? The robot.txt ? How can i check and correct this. My website is discoolver.com
Link Explorer | | MK-Discoolver1 -
Moz's new Link Explorer displaying the DA marginally less than Site Explorer
Moz's new Link Explorer displaying the DA marginally less than Site Explorer. Old one is showing it 46 while new link explorer is showing the DA as 40.
Link Explorer | | dhananjay.kumar11 -
MOZ doesn't work for .dating and .chat domain extensions
I have been a MOZ subscriber for a few years now. I don't think MOZ works for .dating and .chat domain extensions. I have 2 sites that have authority 1 despite back links. Here are the details: https://oooo.dating > DA = 1 https://talk.chat > DA = 1 oooo has 221 links (Google search console) talk has 1317 links (Google search console) May be a MOZ staff member can look into this. If you are customer and use some of the newer domain extensions please share your details if you have the same problem.
Link Explorer | | dmcubed1 -
Am I using OSE wrong?
Hi - new to all this! A link on a competitor site is showing as 404 when I check it with OSE but nothing on the page seems to be broken. It's got loads of links to it, so I'm trying to work out what's going on. I just filtered the results by '40?'. Thanks
Link Explorer | | razor20100 -
Moz Spam Score
Hi! The spam score for my sites is "--" with no graph. The sites are large ecommerce sites with a ton of branded search and DAs of 50+. Simple question - are these clean spam scores, or has Moz not calculated the scores yet?
Link Explorer | | AMHC1 -
How can Ahrefs be better than OSE ?
Dear Sirs and Contributors, I know this question is going to cause revolution, but I am paying for a subscription to moz every single month and I see that Open Site Explorer doesn't find baacklinks for some of my new clients when aHrefs can find more than 16 !! Why is this ? How can a free tool like ahrefs give me more information than OSE for which I pay for ? ;-(
Link Explorer | | Tintanus1