Have Questions about the Jan. 27th Mozscape Index Update? Get Answers Here!
-
Howdy y'all. I wanted to give a brief update (not quite worthy of a blog post, but more than would fit in a tweet) about the latest Mozscape index update.
On January 27th, we released our largest web index ever, with 285 Billion unique URLs, and 1.25 Trillion links. Our previous index was also a record at 217 Billion pages, but this one is another 30% bigger. That's all good news - it means more links that you're seeking are likely to be in this index, and link counts, on average, will go up.
There are two oddities about this index, however, that I should share:
The first is that we broke one particular view of data - 301'ing links sorted by Page Authority doesn't work in this index, so we've defaulted to sorting 301s by Domain Authority. That should be fixed in the next index, and from our analytics, doesn't appear to be a hugely popular view, so it shouldn't affect many folks (you can always export to CSV and re-sort by PA in Excel if you need, too - note that if you have more than 10K links, OSE will only export the first 10K, so if you need more data, check out the API).
The second is that we crawled a massively more diverse set of root domains than ever before. Whereas our previous index topped out at 192 million root domains, this latest one has 362 million (almost 1.9X as many unique, new domains we haven't crawled before). This means that DA and PA scores may fluctuate more than usual, as link diversity are big parts of those calculations and we've crawled a much larger swath of the deep, dark corners of the web (and non-US/non-.com domains, too). It also means that, for many of the big, more important sites on the web, we are crawling a little less deeply than we have in the past (the index grew by ~31% while the root domains grew by ~88%). Often, those deep pages on large sites do more internal than external linking, so this might not have a big impact, but it could depend on your field/niche and where your links come from.
As always, my best suggestion is to make sure to compare your link data against your competition - that's a great way to see how relative changes are occurring and whether, generally speaking, you're losing or gaining ground in your field.
If you have specific questions, feel free to leave them and I'll do my best to answer in a timely fashion. Thanks much!
p.s. You can always find information about our index updates here.
-
Thanks Matt I'm proud of the team's work on growing the index thus far. I think we've reached the top of where we can go with the current index's infrastructure, so I'd expect sizes will stay in this range for the next 5-6 updates at least.
For the last 4 years, we have been working on a new infrastructure for our indices - something closer to what Google does with real-time processing via caffeine (though not quite as robust), and we're planning to launch that in Q4 of this year, at which time, our index can grow much bigger and much faster (it'll also be fresher, included lots more kinds of data, etc). That system also won't be limited by software (which holds us back today), but rather by hardware (which we can and will buy more of). I really can't wait for that
-
I second that opinion, super exciting to get deeper information! Can't wait to dive in!
-
I don't have a specific question, just a WOW! I remember when the index was getting smaller & smaller as you guys went through some "figuring out" of how exactly you would index the whole internets. It has come back in SUCH a big way!
I was thinking of your "false narratives" /rand blog post and how things didn't always go the way you wanted. OSE's limits have always been one of those "not the way I wanted it to be" with Moz and this size of update is an AMAZING comeback.
So, no question - just a bit of a "great job, team!" to get the index to this size. Can't wait for EVEN MORE.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
API v1 still gets new data?
Hello! Do the v1 API endpoints provide fresh data, or do I need to use the v2 endpoints for fresh data? According to the v1 API docs "This guide outlines the endpoints for now archived Mozscape API endpoints." Does this mean that the v1 API only serves archived data? Thanks!
API | | peterkovacs0 -
Sitemaps and Indexed Pages
Hi guys, I created an XML sitemap and submitted it for my client last month. Now the developer of the site has also been messing around with a few things. I've noticed on my Moz site crawl that indexed pages have dropped significantly. Before I put my foot in it, I need to figure out if submitting the sitemap has caused this.. can a sitemap reduce the pages indexed? Thanks David. TInSM
API | | Slumberjac0 -
To get top searched from the website
I need to get top searched keywords by user from the given website URL through API call. Please help me to achieve this. I get anchor text and link metrics but no documentation to get Keywords. Thank in advance.
API | | Darwill0 -
Get All Campaigns
Hi , i can not find in api documentation how can I browse all campaigns in account in PHP. I need browse Tracked Keywords Overview table for every campaigns. But in documentation i can not find how put together query parameters. In response i need keyword, rank and URL for one project and every project in Campaigns. Is it possible ? Thank you for your answer.
API | | Sabanero0 -
What happens if I go over my Mozscape api free limits?
Hello,
API | | FPK
I just started using the free version of Mozscape and I fully understand there are limits and charges under this category. However to avoid any costly surprises, I like to know:
What happens when I get near my usage limit?
What happens when I just hit the limit?
What happens when I past the limit? Along with my questions, Is there any alert systems to let me know when I get to the range of said limit, like an email?0 -
Domain Authority Decreased on May 5 Update
Why the Domain Authority of my website decreased? Last month my website www.i-phony.com domain authority was 32/100. but today I see the result is only 23/100 . What was the reason behind this big decrease !! Your answer is highly appreciated and very valuable for me. Best regards
API | | iphony0 -
Can we get access to Moz's Rank Tracker via the API?
I'd like to be able to pull the results from Rank Tracker into my own application. Can I access it via an API? I don't see it anywhere in the Moz documentation, which is usually a clear answer. If not, how do you suggest to automate the inclusion of this data without, for example, being blacklisted?
API | | MB070