August 3rd Mozscape Index Update (our largest index, but nearly a monthly late)
-
Update 5:27pm 8/4 - the data in Open Site Explorer is up-to-date, as is the API and Mozbar. Moz Analytics campaigns are currently loading in the new data, and all campaigns should be fully up-to-date by 4-10pm tomorrow (8/5). However, your campaign may have the new data much earlier as it depends on where that campaign falls in the update ordering.
Hey gang,
I wanted to provide some transparency into the latest index update, as well as give some information about our plans going forward with future indices.
The Good News: This index, now that it's delivered, is pretty impressive.
- Mozscape's August index is 407 Billion URLs in size, nearly 100 Billion (~25%) bigger than our last record index size. We indexed 2.18 trillion links for the first time ever (prior record was 1.54 trillion).
- Correlations for Page Authority have gone up from 0.319 to 0.333 in the latest index, suggesting that we're getting a slightly more accurate representation of Google's use of links in rankings from this data (DA correlations remain constant at 0.185)
- Our hit ratio for URLs in Google's SERPs has gone up considerably, from 69.97% in our previous index to 78.66% in the August update. This indicates we are crawling and indexing more of what Google shows in the search results (a good benchmark for us). Note that a large portion of what's missing will be things published in the last 30-60 days while we were processing the index (after crawling had stopped).
The Bad News: August's index was late by ~25 days.
We know that reliable, consistent, on-time Mozscape updates are critically important to everyone who uses Moz's products. We've been working hard for years to get these to a better place, but have struggled mightily. Our latest string of failures was completely new to the team - a bunch of problems and issues we've never seen before (some due to the index size, but many due to odd things like a massive group of what appear to be spam domains using the Palau TLD extension clogging up crawl/processing, large chunks of pages we crawled with 10s of thousands of links which slow down the MozRank calculations, etc). While there's no excuse for delays, and we don't want to pass these off as such, we do want to be transparent about why we were so late.
Our future plans include scaling back the index sizes a bit, dealing with the issues around spam domains, large link-list pages, some of the odd patterns we see in .pl and .cn domains, and taking one extra person from the Big Data team off of work on the new index system (which will be much larger and real-time rather than updated every 30 days) to help with Mozscape indices. We believe these efforts, and the new monitoring systems we've got will help us get better at producing high quality, consistent indices.
Question everyone always asks: Why did my PA/DA change?!
There are tons of reasons why these can change, and they don't necessarily mean anything bad about your site, your SEO efforts, or whether your links are helping you rank. PA and DA are predictive, correlated metrics that say nothing about how you're actually performing. They merely map better than most metrics to Google's global rankings across large SERP sets (but not necessarily your SERPs, which is what you should care about).
That said, here's some of the reasons PA/DA do shift:
- The domains/pages with the highest PA/DA scores gain even faster than most of the domains below them, making it harder each index to get higher scores (since PA/DA are on a logarithmic scale, this is smoothed out somewhat - it would be much worse on a conventional scale, e.g. Facebook.com 100, everyone else 0.0003).
- Google's ranking algorithm introduces new elements, changes, modifies what they care about, etc.
- Moz crawls a set of the web that does or doesn't include the pages that are more likely to point to a given domain than another. Although our crawl tends to be representative, if you've got lots of links from deep pages on less popular domains in a part of the web far from the mainstream, we may not consistently crawl those well (or, we could overcrawl your sector because it recently received powerful links from the center of the web).
My advice, as always, is to use PA/DA as relative scores. If your scores are falling, but your competitors' are falling more, that's not a bad thing. If your scores are rising, but your competitors' are rising faster, they're probably gaining ground on you. And, if you're talking about score changes in the 1-4 points range, that's not necessarily anything but noise. PA/DA scores often shift 1-4 points up or down in a new index so don't sweat it!
Let me know if you've got more questions and I'll do my best to answer. You can also refer to the API update page here: https://moz.com/products/api/updates
-
Rand, I've emailed you. Thx
-
Where are you seeing that? In OSE? Or in Moz Analytics? In Moz Analytics, it's possible that it's still cached, and will be updating (a few thousand campaigns each hour, so not too long until all of them are done), but in OSE, that data should absolutely be new. If not, can you send an email to me - rand at moz dot com - with your sites, and I'll ask the Big Data team to look into it.
-
Hi Rand, I'm still seeing 9 June in my campaigns and no updated data....or missing data. Not fixed here yet.
-
Yup - I'm seeing the same team. Have let our engineers know - hopefully they can sort it out and fix soon.
-
Rand, I'm seeing some seriously weird data on many of our sites. Crazy Euro links that go nowhere...that definitely aren't meant to be there, and link totals that don't add up.
-
I'm seeing some odd ones too that appear not to have updated. Pinging the team as it shouldn't usually take this long for data to update.
-
Update:
some of the sites we are tracking have data in them but it's still from 9 June. The rest are showing incorrect / corrupted links or no links at all.
Conclusion: there is something seriously wrong with the MozScape update for us.
-
Hey Sticky! It takes about 24-48 hours for new index information to be submitted to Moz Campaigns anytime a new Mozscape Index is released. By checking your domain directly on OSE (moz.com/researchtools/ose) you will be able to see your data—and more—before campaigns are updated. This may be slightly delayed as we are building monthly data for all campaigns which we run on the 1st of each month. Generally our index updates are rarely released near the beginning of the month which would not interfere with normal campaign updates.
Hope this helps and let me know if you have any questions!
-
It is working on most sites, but a few I have just checked have changed, ie one started at 27 - 5 hours ago was 32 now 30! So might give it another 24 hours to settle down.
-
Glad it is working for you. I'm still seeing last Index, and in some cases no data.
-
I had to re-fresh a few pages, a few times, but all the data has come though now. Every website up, though a few by only 4, but I am still hopeful that is not noise but the result of hard work.
-
I'm wondering why I can't see the updated MozScape data in my account? It still says next index 9 June and the data still appears to be old (and / or incomplete). Any advice?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Regarding Moz API token password update
Hi, In March we have updated password for MOZ API and used in our application it worked, but currently the updated password is not working and in the MOZ site the old password is shown and its active. We are using Legacy username and password.
API | | NickAndrews
We see that 5 tokens can be added for API, if we add 2 tokens both will be active.
We are currently using free services. Please help us resolve this issue.0 -
Unsolved Mozscape API subscription
We have questions regarding our subscription and the plan we are on. We are more interested in Mozscape API and not the features that we have access to currently. Will you let us know how we can change. Is there some one that we can chat with. Thanks,
API | | PatientPop
Naveen
naveen.sarabu@patientpop.com0 -
Website domain authority dropped from 55 to 1 in two months
Hi all! Our website DA dropped from 55 to 1 in a matter of two months. Last time I checked, it was two months ago and the website DA was 55. When I checked today, it is 1. Needless to day, I am shocked. The website has been around since 1995 and has always had pretty decent DA and PA. We made a couple of important changes during past 3-4 months. One is to support HTTPS and another is to implement the responsive web design to support browsing by mobile devices. What could be the reason for such a dramatic drop of DA in a short period of time? Where would you suggest me to look for it? How can we get the problem fixed? Here is the website URL: www.infohub.com Thank you!
API | | jimz80701 -
Sitemaps and Indexed Pages
Hi guys, I created an XML sitemap and submitted it for my client last month. Now the developer of the site has also been messing around with a few things. I've noticed on my Moz site crawl that indexed pages have dropped significantly. Before I put my foot in it, I need to figure out if submitting the sitemap has caused this.. can a sitemap reduce the pages indexed? Thanks David. TInSM
API | | Slumberjac0 -
Is everybody seeing DA/PA-drops after last MOZ-api update?
Hi all, I was wondering what happend at the last MOZ-update. The MOZ-health page has no current errors and i've checked a lot of websites probably about 50 a 60 from our customers and from our competitiors and everybody seems to have a decrease in DA's / PA's. Not just a couple went up and down like normally would happen, but all seems to have dropped.... Is anyone seeing the same in their campaigns? Greetings,
API | | NielsVos
Niels3 -
What happens if I go over my Mozscape api free limits?
Hello,
API | | FPK
I just started using the free version of Mozscape and I fully understand there are limits and charges under this category. However to avoid any costly surprises, I like to know:
What happens when I get near my usage limit?
What happens when I just hit the limit?
What happens when I past the limit? Along with my questions, Is there any alert systems to let me know when I get to the range of said limit, like an email?0 -
Suggestion - Should OSE include "citation links" within its index?
This is really a suggestion (and debate to see if people agree with me), with regard to including "citation links" within Moz tools, by default, as just another type of link NOTE: when I am talking about "citation links" I am talking about a link that is not wrapped in a link tag and is therefore non clickable, eg moz.com Obviously Moz have released the mentions tool, which is great, and also FWE which is also great. However, it would seem to me that they are missing a trick in that "citation links" don't feature in the main link index at all. We know that Google as a minimum uses them as an indicator to crawl a page ( http://ignitevisibility.com/google-confirms-url-citations-can-help-pages-get-indexed/ ), and also that they don't pass page rank - HOWEVER, you would assume that google does use then as part of their alogrithm in some manner as they do nofollow links. It would seem to me that a "Citation Link" could (possibly) be deemed more important than a no follow link in Googles alogrithm, as a "no follow" link is a clear indication by the site owner that they don't fully trust the link, but a citation link would neither indicate trust or non trust. So - my request is to get "citation links" into the main link index (and the Just Discovered index for that matter). Would others agree??
API | | James770