How to download full list of internal links properly with OSE?
-
Hi Moz Community,
I am having issues getting a full list of my internal links when I go to the Open Site Explorer. When I export the internal links into excel, I am getting half of my URLs and 80% of what I am getting is absolutely crazy URLs that are super super old.
Plus, these old URLs are not even showing in my campaign crawls. So the crawls themselves are different.
Any help on this would be greatly appreciated, since I am trying to get a full picture on the internal link structure of my site.
Thanks!
-
Agree, I love Screaming Frog!
-
Hi Ryan,
It would be best to use Screaming Frog if you want a true indication of your internal link structure.
https://www.screamingfrog.co.uk/seo-spider/
Cheers,
David
-
Hey there! Tawny from Moz's Help Team here. I think I can help explain why you're not seeing all the links you might expect, and why you're seeing some older links that you're not seeing in your main Campaign Site Crawl.
Open Site Explorer and the Link Analysis page of Moz Pro Campaigns are both tied to our Mozscape index, which tends to update roughly once a month.
Just a few points on how we compile our index:
-
We grab the most recent index.
-
We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
-
We start crawling from the top down until we've crawled ~130 billion URLs
The idea here is that we're focusing on the highest-quality links we can find, coming from the most prominent pages of authoritative sites. So, while you may not see every link for a site within our index, we're aiming to report the most valuable ones available!
Most new sites and links will be indexed by our spiders and available in Mozscape and Open Site Explorer within 60 days, but some take even longer for many reasons, including the crawl-ability of sites, the number of inbound links to them, and the depth of pages in subdirectories. This tends to bias our index in favor of newer links. Linking data is only stored in the index for about 180 days, so unless the crawler has a compelling reason to return to your site and the sites linking to yours and rediscover those links, they can fall back out of our index again. That doesn't mean they're not out there affecting your SEO, just that our tools don't see them anymore.
You can see our most recently updated schedule here as well as some more technical metrics on our Mozscape API Updates page. You can also see when the last and next updates happened on the Open Site Explorer (OSE) homepage at any time.
Since Moz focuses on quality of links over quantity, we are always focused on the most relevant links to display to our users. It's possible that Moz's index will leave out some of the lower-quality (non-link juice providing) links out of our index because of this. So, that might explain why you may see some discrepancies with what other tools may be showing.
You can read more about how we build our index in our guide here.
I know this is a ton of information, so if you have any questions or if I didn't make anything clear enough, please don't hesitate to ask! You can always drop us a line at help@moz.com and we'll do our best to clear up any questions you might have.
Cheers! -
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Link tracking list not finding my links
I am using the Link Tracking List to track new backlinks that I have achieved. However, the report still shows about half of them as not found... even some that have been there for several weeks. I thoroughly tested a couple of them to make sure the links were coded in html, there were no nofollow/noindex rules, the bot was not blocked etc, but still found nothing. What can I do now?
Link Explorer | | Zambezikid0 -
How can I get links indexed in Just Discovered Links, or Open site Explorer? (Because it aint Bit.ly or Feedly!)
Dear Moz & Forum, I've been on a bit of a merry-go-round with one of my clients. He knows enough about SEO to ask questions, and one thing he likes is to see the links that we are building inside Open Site Explorer. Now, of course we know that it takes a while for links to show, but that was ok, because we could just plug the links that we've built into a bit.ly link and then they'd show under Just Discovered Links in Open Site Explorer, and then boom, we're happy. It stopped working some time ago, and after the support team were a little unstuck I sent a hopeful tweet to @randfish. Here is the response: https://twitter.com/randfish/status/788412272129347584 The advice to use Bit.Ly was removed swiftly after this! Oops! The current Guidance is here: https://moz.com/help/guides/research-tools/open-site-explorer/justdiscovered It says: Want to see your new link in the Just-Discovered Links tab? An easy way to start seeing your link in the Just-Discovered section is to have it shared on Twitter and available on Feedly link. Feedly stopped supporting their own link shortner a few years ago. Also, when you share a link from Feedly on twitter it is just the same as sharing a normal tweet, because it shares the standard URL for the site not a link to view on Feedly. Or, maybe these two statements are independant of one another and they should both be done separately? I've done that too! I've also added the feed to recommended feeds inside fresh web explorer and that didn't work either! So, Moz and Help team please help me! How can I get links I've built for clients into Just Discovered Links? Thank you!
Link Explorer | | stevestretton11 -
Get a List of Pages in a Site, Ranked by their PA?
Is there a way to get a list of the pages within one of my sites, and rank it by page authority?
Link Explorer | | JDigitalIdentity1 -
No Competitive Link Metrics Published Since October 28th, When Are They Expected?
MOZ has not published competitive link metrics sinceOctober 28th. Does any one know when we can expect to see them published? Thanks,
Link Explorer | | Kingalan1
Alan1 -
Linking C Blocks
Hello everyone! Just looking to get more information in regards to Linking C Blocks in layman's terms? We have dropped from 10 to 2 since the latest Moz Open Explorer update and are concerned about this score. Our highest competitor has doubled what they had on the last update. Anyone who can provide more information about this, we would greatly appreciate it. Thanks ahead of time!
Link Explorer | | Essential-Pest0 -
Discrepancies between Mozbar data and OSE?
Hey everyone, I've been doing some research into keyword competition, and i've come across a few instances where the data that Mozbar provides differs pretty greatly from what open site explorer is telling me. An example: For this site: http://www.bcwsupplies.com When I have it returned in the serps, Mozbar is telling me there are 21,306 links & 218 linking root domains. But, when I check the backlink profile via OSE, it's only showing 3,020 links & 88 root domains. Can anyone shed some insight into this? Thanks!
Link Explorer | | RCDesign740 -
Total links
Hi there, Root domain metrics I do not understand the "Total links" metric. I was checking this domain: domesticviolenceuk.org. The total links value (11k) is huge compared to the sum of external and internal links (429 + 221). Why? Is that because 221 is the number of internal links to the single page and not to the root domain? If so, it would mean this domain has 10k internal links. Thank you. Willy
Link Explorer | | WillyBin0 -
Inbound Links Sorted by Linking Domain
Hi All -- Does anyone know if it is possible to generate a report (or is there a setting) to have the OSE inbound links report sorted by OSE linking domain? Ideally, the report would create a second column in the "Inbound Links" report with "Linking Domain" and then you could create a nice report that cross-tabbed the links. While you could do some excel text manipulation to do this, does anyone know if it's possible to do through the report generator? Thanks, Matt
Link Explorer | | Resolution_Rentals0