Spring is here and so is our May Index Update!

[IanWatson]

Happy Index Release Day!

For the second month in a row, our hard-working, supremely dedicated Big Data team has delivered our Index Update EARLY! Beyond being punctual, the May Index is one of our most comprehensive and largest update of the year for Moz.

Let’s dig into the details:

• 162,225,495,455 (162 billion) URLs.

• 1,135,327,420 (1.1 billion) subdomains.

• 194,346,505 (194 million) root domains.

• 1,168,465,575,815 (1.1 Trillion) links.

• Followed vs nofollowed links

• 2.84% of all links found were nofollowed

• 65.80% of nofollowed links are internal

• 34.20% are external

• Rel canonical: 28.89% of all pages employ the rel=canonical tag

• The average page has 92 links on it

• 76 internal links on average.

• 16 external links on average..

Go have fun with your new data!

PS - For any questions about DA/PA fluctuations (or non-fluctuations) check out this Q&A thread from Rand: https://moz.com/community/q/da-pa-fluctuations-how-to-interpret-apply-understand-these-ml-based-scores

[julie-getonthemap]

Thank you for this update. I have Tumblr, Blogspot etc. that I use as part of my strategy and it is quite disconcerting this morning to see an individual post on a Tumblr blog with a PA of 94.  It's like looking in your bank and seeing a balance of 100K.

[randfish]

Thanks Ian. I wanted to add on that we've got a little bit of strange issue with some domains in this index that may affect some sorting and link lists.

Basic story: We blacklist domains that have more than 10,000 subdomains and have other features that we've observed make them, almost always, webspam. Since they can adversely impact metrics and scores, we just don't include them in the index. The last 6 months, this process has gone really well, but this index, we hit a bug, and accidentally excluded ~750 domains that aren't all spam, including big ones like Etsy.com, Wikia.com, Blogspot.com, Quora.com, and others.

Initially, folks noticed because the PA/DA scores for these sites were all 1. That part has been fixed, and the metrics should now be more accurate. But, we don't have the link lists for these sites, and we don't have PA scores for the interior pages on these domains. As such, if you're sorting your link lists or if you're researching links that point to a page on Etsy.com or Blogspot.com or Wikia.com or a few others, you'll see no data. This will be fixed in our next index, but we couldn't fix it in this one. The bug was caught, but caught too late, and thus the next 28(ish) days will have some of these sorting and link list issues on these ~750 domains (of which, I estimate, only ~150 or so are actually legit sites - most of the others are, indeed, spam).

If you've got any additional questions I can answer about this, please let me know. Apologies for the error from me and from the team - we'll make sure to get this right in the future, and we've set up new kinds of tests to prevent any similar issues from going unnoticed.