How to download full list of internal links properly with OSE?
-
Hi Moz Community,
I am having issues getting a full list of my internal links when I go to the Open Site Explorer. When I export the internal links into excel, I am getting half of my URLs and 80% of what I am getting is absolutely crazy URLs that are super super old.
Plus, these old URLs are not even showing in my campaign crawls. So the crawls themselves are different.
Any help on this would be greatly appreciated, since I am trying to get a full picture on the internal link structure of my site.
Thanks!
-
Agree, I love Screaming Frog!
-
Hi Ryan,
It would be best to use Screaming Frog if you want a true indication of your internal link structure.
https://www.screamingfrog.co.uk/seo-spider/
Cheers,
David
-
Hey there! Tawny from Moz's Help Team here. I think I can help explain why you're not seeing all the links you might expect, and why you're seeing some older links that you're not seeing in your main Campaign Site Crawl.
Open Site Explorer and the Link Analysis page of Moz Pro Campaigns are both tied to our Mozscape index, which tends to update roughly once a month.
Just a few points on how we compile our index:
-
We grab the most recent index.
-
We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
-
We start crawling from the top down until we've crawled ~130 billion URLs
The idea here is that we're focusing on the highest-quality links we can find, coming from the most prominent pages of authoritative sites. So, while you may not see every link for a site within our index, we're aiming to report the most valuable ones available!
Most new sites and links will be indexed by our spiders and available in Mozscape and Open Site Explorer within 60 days, but some take even longer for many reasons, including the crawl-ability of sites, the number of inbound links to them, and the depth of pages in subdirectories. This tends to bias our index in favor of newer links. Linking data is only stored in the index for about 180 days, so unless the crawler has a compelling reason to return to your site and the sites linking to yours and rediscover those links, they can fall back out of our index again. That doesn't mean they're not out there affecting your SEO, just that our tools don't see them anymore.
You can see our most recently updated schedule here as well as some more technical metrics on our Mozscape API Updates page. You can also see when the last and next updates happened on the Open Site Explorer (OSE) homepage at any time.
Since Moz focuses on quality of links over quantity, we are always focused on the most relevant links to display to our users. It's possible that Moz's index will leave out some of the lower-quality (non-link juice providing) links out of our index because of this. So, that might explain why you may see some discrepancies with what other tools may be showing.
You can read more about how we build our index in our guide here.
I know this is a ton of information, so if you have any questions or if I didn't make anything clear enough, please don't hesitate to ask! You can always drop us a line at help@moz.com and we'll do our best to clear up any questions you might have.
Cheers! -
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Local listings aren't being crawled?
Hello, SEO brainiacs. Here is the problem I can't figure out:We have a lot of clients and all of them have local listings, such as Insider Pages, Yahoo local, Bing locals, Yelp, Yellow Pages etc etc etc.
Link Explorer | | seomozinator
And all of those local listing have company's website link. But none of those links are considered (found) in reports, such as opensite explorer or in Link Analysis Section of Moz PRO. I do know that MOZ crawl considers only high DA websites, but all of local listing domains have above 80-90 DA score, so they can't be NOT crawled. I will say that some local listings do appear on very few reports, but far not all of them. So, what can be the problem?0 -
OSE Inbound Link Report Exports Returning Blank
We're trying to download a larger OSE report of inbound links and the report only has headers and no actual data. I'm logged in, so is there a bug here?
Link Explorer | | Highland0 -
Just Discovered links - How long before they are counted toward the PA scores
I have a new website I put up in the last 90 days. I have been watching the DA, PA and Keyword rankings. Once I set up the campaign I got a report back a week later and some data showing some keyword rankings and some DA & PA scores and links showing from 2 domains. Since this time OSE has Just Discovered 16 more backlinking domains - however the DA and PA scores have not changed ? How long after the new Just Discovered backlinks before they affect the scores I am seeing for my DA & PA ? Thank you in advance for any insight on this matter, Happy Holidays - Joe
Link Explorer | | jlane90 -
OSE - Link Opportunities
I'm checking out this new Link Opportunities feature for one of our sites and what I'm seeing right now is pretty disappointing. For Reclaim Links, everything listed is an internal link on our site. It's got tons of URLs from our old Iciniti structure (replatformed to Magento mid-March). It's crawling tons of stuff that's blocked in robots. There are no links from external domains in the first 5 pages. For Unlinked Mentions, it's showing tons of mentions - 33,566 to be exact, a ridiculous number - mostly from news sites like Forbes, WSJ, Guardian, CNN, etc. These sites are not mentioning us. It's set up to look only for our brand name or domain name, so I don't know how it's thinking there are all these nonexistent mentions. What's going on with Link Opportunities?
Link Explorer | | Kingof50 -
Why does Open site explorer only show a fraction of the linked Domains that Google does?
One of my sites ( http://www.georgerossphotography.com ) shows 19 root domains and 155 links on Open Site Explorer. This doesn't pass the sniff test with me and when I look at Google Webmasters they show 120 domains with 1,870 links. I am assuming that Google use it's own data for ranking purposes which makes me question the validity of Open Site Explorer? Am I missing something ? I have been using OSE as my primary tool and now I feel that it has little value. I would appreciate any feedback. George.
Link Explorer | | sirgeorge0 -
Why won't Open Site Explorer detect inbound links from big name websites?
I understand smaller, spammy sites woudn't get indexed, but some of the websites linking to mine have high domain authorities. The website is a parked subdomain.
Link Explorer | | myers3680 -
Identifying Recently Added Links?
Is there a way to see specifically which links were added in my campaign? I noticed that this week we have 133 new links that were added to out website, yet we lost 1 link on the root level. So I am a little confused where that is coming from. I wanted to see if I could see where. When I look at the recent inbound links in the OpenWeb explorer there isn't anything listed. Thanks!
Link Explorer | | HashtagHustler0 -
Root Domain and Subdomain Links OpenSite Explorer
I'm using opensite explorer trying to explore who is linking to my clients site. In the main blue area it shows total links as 20, which I understand is to the specific page. In the Root Domain Metrics it shows 4,032 external links. Sub domain Metrics shows 1,973. How do I find out where the links are? I tried creating a advanced link report with the "Links that to" are checked off as "any page on this root domain" and only shows 91 of the links? What am I doing wrong or missing?
Link Explorer | | JoshBowers20120