301 Redirects: How long for Google to recognize? How long for Moz/OSE to recognize?
-
I redirected a few pages that had funky URL issues (capitals and underscores and useless words) to the same page renamed with proper URLs. The sitemaps were changed also, and re-submitted.
Like this: mydomain.com/Not_Pertinent_Words.html >>> mydomain.com/good-words.html
Google seems to have found them and changed the search engine results listing in about 8 days.
But it's been about a month now and Moz and OSE still have not transferred all the strength and link data from the old URLs to the new ones.
Question 1: How long does it usually take Google to transfer all the link and strength data for a 301?
Question 2: How long does it take Moz and OSE to do the same? Is there something I need to do to tell them about the changes?
-
I had a similar issue with usedcars.com. We had several hundred thousand URLs that were redirecting from parameter driven URLs, to clean hierarchical URLs with a huge initiative to clean them all up in a few months. It took well over a year and still didn't see them getting recognized.
I blogged the process - but the part about what I did to speed it up:
I grabbed one of my many impulsive purchased domains and quickly set up hosting and an old school html site that consisted of one page. I then exported all of the links on the Google "site:" search through a Firefox plugin called SEOquake that exports the results into a csv file. It's not the prettiest, and there was a lot of work still needed to get to just the URLs, but it was the best solution I could find (note: if any SEO reading this knows of an easier way to do this - please add to the comments for prosperity). I then parsed out the parameters in the URLs in a separate document and used those as the anchor text for each URL. Finally, using excel I then concatenated the URLs and parameters (that were now anchor text) into an html href string.
Then copying and pasting the "string" column into the html code, the page looked like:
The page wasn't the prettiest, and it had thousands of links (the above is just an example) so it was bad all around, but the point was to get those links crawled by Google.Of course every SEO knows that you can't just build a website and expect it to immediately get crawled - right?
So I set it up in Google Webmaster Tools and submitted the page to the the index:
I even got more fancy to ensure Google would see the page and crawl all of those old URLs and +1'd it on Google.Did it work?
I checked the URLs this evening to see how many Google is seeing and the number has dropped from 550,000 to now only 175.
I took the domain off of the server, and now have it parked elsewhere (back where it belongs) and removed the webmaster tools account. All traces of it ever existing are now gone, and the small moment of my attempt to get those URLs removed has passed.
It's very black hat - and could potentially get you in trouble if not approached carefully. But - it worked...
-
Hi Gregory,
Great suggestion. It might be a little challenging to mix the private crawl (Your campaign) with the public one (OSE) but it's certainly possible to cross-pollinate data.
In the meantime, we're working on better incorporating our "just discovered links" in OSE to be included in the main index. This means that anytime a link is tweeted (our source of discovery for Just Discovered Links) it will be included for crawling in the general index.
Regardless, thanks for your input into this matter, and I apologize for it being a frustrating experience.
Cheers,
Cyrus
-
Hi Greg
So our new index went up today. Looking at the last crawl date for www.easydigging.com we crawled the domain on 9/27 but did not crawl deep enough to find the original page containing the 301. So it may be a couple of more indexes before we crawl deeper to find it. What you would need to do help www.easydigging.com increase it's visibility/domain authority which will result in OSE finding more pages which will eventually find the link we need. The current page is no longer in our index as it not longer has any crawl authority and pretty soon it will not show any data in OSE unless it is re-crawled.
Once we can include "Just Discovered Links" into the main index, as Cyrus mentioned above, all you would need to do then is have any link tweeted from a high ranking account which will basically guarantee a crawl to that page.
-
I don't have documentation, but the new page it redirects to has this tag:
So I would guess I did the 301 redirect the to this new page at the same time I launched the new page (there was no major design changes - just shortening the URL and getting rid of the capitals and underscores)
-
Hi Greg
Would you be able to tell me when the 301 was added for http://www.easydigging.com/Garden_Cultivator/wheel_hoe_push_plow.html?
-
Hi Greg
We do set the expectations about how long it would take for us to process pages from our help hub:
Here's how we compile our index:
- We grab the most recent index.
- We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
- We start crawling from the top down until we've crawled 65,000,000,000 pages (which is about 25% the amount in Google's index).
- Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index.
We update our Mozscape Index every 4 weeks. Crawling the entire Internet to look for links takes 2-3 weeks, but our crawlers are always in motion. When we need to start processing, we grab all the data they have collected and start processing which can take up to 3 weeks to determine which of those links are the most important.
So it's very possible that it may be re-crawled with an index we are possibly updating today ahead of schedule. Like I mentioned earlier, it can take a while for metrics to transfer over as it isn't guaranteed that we will crawl the same page for each index update.
-
Thanks David. That does explain the issue better.
However, the occasional really long lag time for OSE to re-crawled pages that have been 301 redirected is still a problem for those of use who subscribe to Moz and are trying to use it to monitor and improve our SEO. But I think there is an easy solution...
Give Moz subscribers a new tool like GWT's "Fetch as Google" - but call it "Fetch as Moz" and have it trigger BOTH the Moz spider and the OSE spider to go re-crawl that page. This would let us quickly and positively know that our data for 301'd pages has been transferred to the new page.
Can this be done?
-
Hi Greg
It can take a while for all of your pages to be picked up in new indexes as I mentioned earlier. OSE will actually check for re-directs immediately but for the example page you provided, that page was last crawled on September 8th which did not have the re-direct at that time. It is not guaranteed that the same pages will be re-crawled during the next index updates which is why our metrics from seomoz.org moved to moz.com took a few months. OSE does not crawl pages in a hierarchal manner unlike our other crawler for diagnostics which follows links within the source code.
OSE needs a reason to re-crawl new and old pages so it can see the re-direct. We take the last 40 days of crawl data from the last Index if a page hasn't been re-crawled. We actually are about to release a new index today ahead of schedule so it's possible we may be able to re-crawl the old page. After I confirm a new index is complete I will re-check the old URL to see if OSE picked it up.
-
Hi David,
Sorry to be blunt here, but no it doesn't make sense - and I also think it is incorrect...
Websites do internal 301 redirects all the time (redirecting from old page to new page within the same site) BUT your explanation above appears to say that OSE does not follow those redirects and move the data from old to new url ? That would make for a pretty useless SEO program...
Your message also appears to say that if we have to go contact everybody that links to www.mysite.com/old-page.html and have them change their link to www.mysite.com/new-page.html ? That's just silly...
I think your explanation is incorrect because SOME of the 301 re-directs I have done on my site have passed the data perfectly in OSE and Moz. Run OSE on these two URLs and you will see: Old URL = www.bestdryingrack.com/clothes_drying_rack.html New URL it was 301'ed to = www.bestdryingrack.com/clothes-drying-rack-main.html
So it appears that inconsistency is the a big problem here. That should be some firm rule like: "Moz will transfer all data across 301 redirects on the same domain within 1 month" This would let us analyze or sites without trying to keep track of silly situations of which new pages have the proper data and which ones we still need to fudge between old and new page data.
I hope this all makes sense...
-
Hi Greg
Our last index update was on the 23rd and the next one is not scheduled until Nov 20th. Keep in mind that not all metrics will carry over, we will still index the site based on strong backlinks. For external links you would need to make sure the linking root domains linking the old pages update their links with the new ones as our crawler does not follow links and only sees what they are as is.
So what our index will eventually update/bring over:
Internal Followed Links
Total Internal LinksUpdating the linking root domains with the new links will help with the external links.
Let me know if that makes sense!
-
Hi David,
I waited another week and just now checked the new and redirected URL again. The data STILL has not transferred over to the new URL.
It's been a long time. This excessive lag time in moving data across re-directs really needs to be fixed.
Personally I can mentally keep track of it, but what about the other SEO programs out there that are using Moz data? They have to experiencing "garbage in - garbage out" problems on any analysis involving re-directed URL's.
Can you check into this and let us know what's up?
-
Hello David, Thanks for jumping in!
The old URL was http://www.easydigging.com/Garden_Cultivator/wheel_hoe_push_plow.html and the new one it 301's to is http://www.easydigging.com/wheel-hoe-hoss.html Please reply back here so I know what the situation is.
Since it appears that Index updates only happen monthly, and it takes a few months for data to fully migrate over to redirected URLs, it would be really nice to change the little message that shows on OSE when an analyzed page is redirected (currently something like "Do you want to see the data for page this redirects to") to something that describes the lag, like.. "It takes a few months for Moz data to transfer to redirected URLs, so be sure to check both the old and the new URL since the one with higher statistics is the accurate one"
-
Hi Greg
When setting up your re-direct not all metrics will transfer over to the new domain. It took us a few index updates to see most of our metrics to appear after we moved from seomoz.org to moz.com.
Would you be able to provide us with the URLs of the old site and the new site so we can make sure our crawler is picking up the 301s?
If you do not wish to share them here you can submit your information through our contact form:
-
Hi again Chris. Mostly I was wanting to use Moz to evaluate my site for SEO reasons and frustrated that I kept having to use "old" page URLs to get data.
Also I have just started to experiment with the Penguin Analysis tool from Virante, and it uses Moz data mixed with data from other SEO programs - so I get corrupt results there whether I input the old URL or the new URL. http://www.penguinanalysis.com/
If they really updated the Mozscape index on the 18th, it seems it should show by now...
-
I wouldn't worry about the Mozscape index not showing your changes. I put up a client site on the 8th and, despite thousands of visitors and some very strong backlinks, there is no data for it on the OSE. Like I said, try Majestic, their tool does show this recent client site and the link data is accurate, so it may display more accurate citation and trust flow (Majestic authority metrics) for your site after the redirect than OSE.
-
Thanks Chris. But unfortunately that makes the situation even worse. I did these 301 redirects weeks ago, and your quote say that there was a Mozscape Index update was a few days ago on October 18th.
I just checked again in OSE and the old page data and strength still has not transferred over to the new page. I have attached a screenshot from OSE showing this.
Any ideas?
-
Question 1: If Google has correctly updated the index, I would imagine that it has redirected the link profile as well.
Question 2:
"Just-Discovered Links index updated continuously. Last Mozscape index update: October 18, 2013. Next Mozscape index update: November 15, 2013" From OSE.
If you're looking for results on a more recent site, try Majestic's Site Explorer. It is in my experience more accurate with newer sites and fresh redirects.
Hope this helps.
Chris
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How Does Moz Spam Score Increase and Decrease?
Hello Moz Community, I'm facing spam score issue on my site General Queen according to MOZ my website spam score is 30% but anybody can explain how can i decrease it? Firstly I disavowed all low-quality and high spam score links (6 months ago) but my spam score was not decreased after that I created some high-quality and 0 spam score backlinks but still, my spam score is same 😞 According to Moz link explorer, my domain backlinks spam score ratio was below; Stats of 14 July 2019 .. Screen shot is attached 1-30% .. 45.3%
Link Explorer | | BloggerSEObd
31-60% .. 32.1%
61-100% .. 22.6% Stats of 13 June 2020 .. Screen shot is attached 1-30% .. 65.1%
31-60% .. 30.7%
61-100% .. 4.2% You can see a huge difference in the backlinks spam score in it. But why my spam score is still 30%? Looking forward to your reply. Thanks & Regards, abu1NiB XGLkr5z1 -
MOZ link explorer is not showing any result
Whenever I am searching in link explore, it is only showing the blanks with default zero value. It was showing data for my website 1 hour ago but now not showing. SYphSNw
Link Explorer | | gb.select4you3 -
Error Message on Moz Crawler
Hi all, Just ran into this issue, when analysing this site. Just got this message when using MOZ "Page Optimisation Error". Anyone know why? It seems to be working fine on other SEO analyser tools. Website is: www.sbpcreativemedia.com.au Thanks in advance! luXS8V5
Link Explorer | | Dushala0 -
After how long Moz show the matrices of a new domain
I want to know the Matrices of this site " Clash of lights ". Its a new domain how can i find its matrices like DA PA etc in Moz. Please guide me through it.
Link Explorer | | Muhammadahamd0 -
Different results when running spam score tests with and without www in moz
Hey, can anyone help with this? I am receiving extremely different results running spam score tests with and without www in moz.
Link Explorer | | DarkoA
No www - is getting a spam score 8
And with www. is 1. Should I be worried here, and how should I move to handle this? Thanks in advance guys1 -
Why doesn't Moz crawler follow robots.txt?
It is crawling the entire site, and there is stuff we do not want it to. Please advise.
Link Explorer | | Tylerj0 -
How long does it take for moz to index new domain?
Hi, Two months ago we changed our site www.jicht.nl to jicht.nl. Everthing is redirected properly but the page authority remains 1. How long does is take rogerbot to index the site without www? Google rankings are also dropping so there might be a bigger issue. Thanx
Link Explorer | | AlkaVitae0 -
Expired domains and OSE?
If a domain expires will it remain in OSE indefinitely and show external backlinks? Or at some point does the domain get removed from OSE?
Link Explorer | | lsilver0