Why does Open site explorer only show a fraction of the linked Domains that Google does?
-
One of my sites ( http://www.georgerossphotography.com ) shows 19 root domains and 155 links on Open Site Explorer. This doesn't pass the sniff test with me and when I look at Google Webmasters they show 120 domains with 1,870 links.
I am assuming that Google use it's own data for ranking purposes which makes me question the validity of Open Site Explorer?
Am I missing something ? I have been using OSE as my primary tool and now I feel that it has little value.
I would appreciate any feedback.
George.
-
Hi George - gotcha. That makes sense. The explanation is really simple - we crawl as much as we're able to process (processing and calculating metrics like PA/DA/MozRank/MozTrust/etc are our bandwidth barreirs right now). Other services don't calculate those metrics, but Google & Bing have found ways to do both (large crawls and immediate processing at scale). We're moving to a model similar to Google soon, but in the meantime, our index is smaller than others (particularly in certain sections/parts of the deep web, less trustworthy sites, and more spammy areas of the web).
I will nudge the team to get up a page on Mozscape that helps to better explain this.
-
First of all, thank you for taking the time to respond.
I have been actively linkbuilding this past couple of weeks driving my number of links up from 73 to 427 per Hrefs. During this time I have been monitoring all major link tools and was struggling to fully understand the differences and at the end of the day all that really matters is my SERP position.
In this time, I have been using Linkstant and again when I compare that to Just discovered only a fraction of the new links appear in 'Just Discovered'. Again, I am not trying to be negative I am trying to understand how to best use the tools that I have available to me and how to answer the question that i have raised, should a client ask me the same question.
It is not that the MOZ index is not for my needs is that there is no clear explanation on the OSE page with respect to relationship of the data it returns and that of the other similar services. I could demonstrate to a client that they have more links if they use another link analysis service! That is what is frustrating me. So, if you could provide an explanation of how OSE compares to the other indices that would give me something to work with.
Again, thank you for your time,
George.
-
Also I would suggest using all numerous bit of software.
I use ahrefs, Moz, Webmaster tools and about to sign up for Majestic.
They are all good in their own way and have some great resources (Ahrefs now tells you a list of all your links pointing to 404 pages - great report), but they also have their limitations - unfortunately there isn't one software that does everything amazing so you do need to sign up for multiple resources.
-
I agree its not a great topic for a WBF but a page on the website would be great.
Look forward to meeting you at SearchLove London.
Thanks
Andy
-
Hi Andy - I left a reply below. I could do a WB Friday on this, but the topic doesn't lend itself well to the more tactical foci of WB Friday, and it's a bit Moz centric. That said, I will nudge the team to create a page about how Mozscape's index functions and why counts are different from index to index and across different tools.
Thanks for helping out!
-
Hi George - this is a complex problem/issue, but I'll do my best to explain.
First off, Oleg & Andy are correct - Google has 100s of times the infrastructure (and thousands of times the financial and human resources) that Moz does, and their index is quite a bit larger. For example, the latest Mozscape index is ~180 Billion pages, while Google's main index likely contains something between 4-10X that amount at any given time.
However, there's a lot more complexity involved, because sometimes you'll actually see Moz report more links than Google Webmaster Tools. This is because Google Webmaster Tools samples (they might know about far more than 120 domains linking to you) and they're not specific about how they treat pages that are, for example, canonicalized by a redirect or a rel canonical. They also don't specify the frequency of their updates, and this appears to be inconsistent as well - we've seen counts go up, down, and all over the place without much indication of why. Hence, lots of site owners become suspicious about the link counts shown in GWMT. In your case, because you have a limited number, you can likely validate that all of these are real and so the problem isn't as big.
As far as Moz's index goes, while we're not as big we are always growing (last index, for example, was ~140 Billion pages - you can see all the updates here: http://moz.com/products/api/updates), we're consistent in what we'll tell you (the exact count and every link we see), and you can track this over time compared to index size (and compared to competitors). Next year, we are aiming to 3X the current size, so you should see something much closer to Google's scale (it's been an ongoing challenge for some time, but we're nearly there).
That said, totally understand if our index isn't right for your needs. Majestic SEO and Ahrefs may both be good options (though neither of them are quite Google's size/freshness/scale either, and they have their own weaknesses vs. Moz's index on other features).
I'll also bring back your suggestion of a page that talks more directly about the comparison between different indices and why Moz might have more/fewer links than other tools (and than its own prior indices) over time.
Wish you all the best!
p.s. George - in you particular case, you may also want to check the "Just Discovered" tab in OSE, which often shows a lot of new links we've found that haven't yet made it into the index (but will make it into the next one). If you've received lots of new links in the last 30-60 days, it will take a month or so for those to make it into our main index, but they'll show up within hours in JDL.
-
Thank you for you answer (and sorry you have to write it again) but I have been using OSE as the benchmark for my clients sites without digging any deeper (why should I?) and you may think me naive for that but there is no qualification on OSE with respect to the data it offers.
It does not state that there are limitations on data presented or that the may be incomplete... hence, my original question. This means that I have been providing my clients with incomplete/unqualified data based on OSE reports.
OSE states for Linking Domains "Gauge your site's influence. Analyze the root domains that link to the URL or domain you've entered." Where is the asterix explaining that the data is not complete?
I am a little bit miffed just now.
I need data that I can stand behind and not data that I have to explain like a snake oil salesman. At the very minimum I need to understand the validity of the data and that should be clearly explained at the top of the OSE page.
Thank you again for the explanation.
George.
-
PRO membership you are getting a lot more tools, not just OSE. The moz community is probably worth the $99 on its own in my opinion.
OSE limitation - I will let an employee tell you the official definition but they do a great job crawling as much as the web as feasibility possible on there servers - but they do crawl a huge % of the web and expanding this each month.
-
Hi
I've answered this question a few times, wish Rand would do a whiteboard friday on this and explain once and for all.
Google is huge and have a vast amount of severs that crawl the entire web (and they don't pick up on every link), moz even though its great, is relatively small and can only crawl a proportion of the web (usually the bigger sites and the most visited sites) and rand has said before that they keep growing there servers and can crawl further and further, but for what you pay for moz $99 a month for them to backwards engineer Google and go as deep as they do, you would probably need to paying a lot more, so I would happily take missing out on a few of the lower domain links for the relatively low price I pay each month.
So if Rank, can do a WBF on this subject that would be great - then I wouldnt have to keep typing the same answer I could just link to his video (Google doesnt like duplicate content).
Also mention above, if you really want to know all your links I use ahrefs and Webmaster tools, but even these aren't great.
If a new websites gets built today and links to you - you still have to wait for Google, Ahrefs to find this website - if this website doesn't get links from else where then it might take these sites a while to find the new website, so even these services aren't perfect.
-
Thank you.
But if that is the case
-
what is the value of OSE as a marketing tool?
-
Where is the definition of OSE limitations?
If this is correct then i will seriously consider my PRO membership.
Thanks again for the answer.
-
-
Google has more and better servers and a much larger index than Moz/OSE.
If you want to get a better picture of a website's backlinks, run reports from several backlink tools (OSE, ahrefs, majestic, webmeup, gwt)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz's new Link Explorer, including our revamped index and DA/PA scores is now open to everyone!
Hey Moz Community, Link Explorer is now open to the public! Everyone can access it via a subscription or a free Moz ‘Community’ account. As you may know by now, the brand-new Link Explorer tool is primed to replace Open Site Explorer as Moz’s link building and analysis tool. The Link Explorer project is the result of an incredible amount of perseverance and hard work by the team, and we’re proud to be able to finally share it with you — we know it’s going to revolutionize how you approach link building and make your job easier. You can read more about the tool here in Sarah Bird’s announcement post. Because Link Explorer improves on almost every aspect of Open Site Explorer, the metrics have improved, too. That means you’re likely going to see some Domain Authority and Page Authority discrepancies between OSE’s index and Link Explorer’s index. We definitely suggest you use the new DA/PA from Link Explorer, as they’re more accurate and refresh daily rather than monthly, as was the case with OSE’s index. However, we also realize that many of you use these metrics to report to your clients and colleagues, and a sudden change or fluctuation could potentially make your job harder. Which DA is the real DA? The new DA is based on a much larger index that has many improvements, several of which are designed to make the index more like Google’s than ever before. You should consider moving towards the new DA (and the old DA won’t be updated after April 26th 2018, so the sooner the better). While there will be fluctuations as we improve the model and add features to the index, we expect it to remain largely stable and to be a far more accurate picture of a site’s authority according to how it’s seen by Google. Why is Link Explorer’s DA/PA considered better than OSE’s, and which should I trust? The larger link index with improved crawl selection allows us to produce a stronger model that includes a much larger proportion of the web. That being said, DA and PA should always be considered in the context of your competitors. A drop in PA or DA relative to the old OSE is of little concern if your competitors saw similar movement. Is Domain Authority/Page Authority an absolute score or a relative one? Both DA and PA are relative to the Internet as a whole. If Facebook acquired a billion new links, everyone’s PA and DA would drop relative to Facebook. Because of this, it’s always best to look at PA and DA in comparison to your competitors. What does a drop/raise in DA mean in Link Explorer vs OSE? How can I explain this to my clients when I’m reporting it? DA and PA should always be considered in the context of your competitors. A drop or raise in PA or DA relative to the old OSE is of little concern if your competitors saw similar movement. Reporting that your site has moved from a DA of 45 to a DA of 42 doesn’t tell the whole story, but reporting that your site has a DA of 42 while your main competitor moved from a 43 to a 37 shows that, relative to the sites you’re competing against in the SERPs, your site has significantly more authority and ranking power. What’s happening to MozTrust and MozRank and why, and what should I replace those with? The improvements to our DA/PA and Spam Score metrics now now account for more important nuances in helping you determine one site’s ability to rank higher than another. Because they no longer correlate with Google’s ranking model as well as they used to, MozRank and MozTrust are being deprecated for better metrics. Users should rely on Page Authority, Domain Authority, and Spam Score to determine the importance and quality of pages, domains, and links. I have historical data I use to help my clients benchmark their progress. What do I do now that DA is calculated differently? You should annotate any KPI changes referencing the change in DA and PA. However, most importantly, you should compare those changes to your competitors, as this will best show how strong your site’s authority is relative to the sites you’re competing against in the SERPs. We take updating our metrics very seriously, and our last major update to the model was 7 years ago. Users of Domain Authority and Page Authority can expect us to continue to produce steady, reliable metrics for the long haul, and only make changes to these metrics when we believe the benefits dramatically outweigh the stability of the metric. Do you have any questions about the new metrics? Anticipating a tough time reporting changes to clients or bosses? Metrics, features or functionality missing that you would want to see? Let us know in the thread, and we’ll work to find a good answer for you. Hope you enjoy the new Link Explorer product and the amazing new link index powering it. We are very excited to provide this valuable data to our community and customers.
Link Explorer | | IanWatson9 -
Moz Pro Tools Inbound Links
Is there a way to get a date of when a inbound link was created from a external website. And if so is there a way to add that to the Moz Pro tools reports on a csv file.
Link Explorer | | willakawillow221 -
Will Moz move DA from an old domain to the new?
Hi folks, I'm moving a digital magazine of mine, which has a great deal of authority (DA53!) to a new domain, which is much more brandable. I'm curious if Moz ever passes the actual DA/PA numbers to the new domain. Does the tool pick this up, eventually? I know once the domain is switched, DA will show a "1" -- as the domain is brand new, technically. This is NOT a question about authority/link juice being passed, but rather the numbers being displayed by Open Site Explorer. Some authors do look at Moz DA as an indicator of a sites "power" before applying for authorship, so I'm trying to find a way to move to a more brandable domain, while keeping the OSE stats in place. If it's not possible, I'll have to live with it 🙂 Any input? Thanks everyone!
Link Explorer | | avilawebfirm0 -
Domain Authority Dropped to 1 from 13
Hi, Our domain authority was at 13 two months ago and it dropped to 1 one and half month ago, I waited for MozUpdate but still no luck. my question is if my domain name doesn't have proper non www to www redirect, moz tool will show authority correctly? How do I check if redirect is proper for Moz to account it ? website is working fine though, no problem to redirect from non www to www.com website. Regards, Vishal
Link Explorer | | Sepalika0 -
Dofollow link from Moz user profile not showing in OpenSiteExplorer
Hi everyone! I'm was searching my "new" dofollow link from my Moz user profile http://moz.com/community/users/625691 now I have more than 200 MozPoints, but in OpenSiteExplorer is only showing my "nofollow" link of the icon (the "M" icon in the left of the URL in the profile). Why not appears the dofollow link? Thanks!
Link Explorer | | rubenalonsoes1 -
Open Site Explorer doesn't show local directories or citation links?
Why is it that when I run a link analysis for my local business sites I don't see any links from Yelp, Google+, Facebook, any local citations? Regardless of them being follow or no follow isn't the hyperlink to a website from a Yelp profile considered a link? Same with any other local business directory? I know the company is listed on a few dozen because I can see it manually but it's not showing up in Moz? Why is it that all local directories and citations with website are omitted? And since it is not there, where is a good place to go to discover these things? I have tried whitespark. It was okay but honestly I wasn't terribly impressed. The data was good, the UI was just okay, but with smaller clients it was an initial check and then very occasional checks that needed to happen. We would go months without using it.
Link Explorer | | bricegump0 -
Repeated mysterious 404's from ancient site structure killing my rankings
Several years ago I changed my site structure to go from a flash based site to a blog based wordpress site. After doing so I went from page 1 to page 30 for my relevant search terms. I have employed people to help me track down the problem and I believe that they have narroed it to the existance of 404's being created from some unknown internal source. I have been for years getting links like this... <colgroup><col width="792"></colgroup>
Link Explorer | | dfphotographer.com
| http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/?share=facebook http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/charisma-and-steve-301/?share=email http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/photography-brisbane-04-2/?share=email http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/photography-brisbane-12-2/ http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/photography-brisbane-13-2/ http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/photography-brisbane-13-2/?share=facebook http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/photography-brisbane-13-2/feed/ http://www.dfphotographer.com.au/brisbaneweddingphotographer/2011/10/brisbane-wedding-photographer-charisma-and-steve-victoria-park-brisbane/photography-brisbane-16-2/?share=email | ......regularly showing in webmaster tools, (this is from a top pages report from MOZ where there are hundreds also shown). When I do a moz crawl of the site, none of these links show up. Therefore I have no way of finding the source of these links (they also do not show me the source in WMT as they should). We have completely cleared the site and rebuilt it and although it is still only a couple of weeks in it still does not appear to have stopped them. Does anyone have any way of helping me find the source of these mysterious 404's?0 -
Domain Authority Trending Down?
Hi All, Can anyone (preferably a Moz representative) explain why Domain Authority has been trending down on multiple domains? I suspect there is a new index in place? And / Or algo changes? This is making life very difficult for us as our client www.cleanawater.com.au has been working very hard to implement only the whitest of white SEO models.
Link Explorer | | Digital3600