Only one internal Equity Passing Link
-
Our web site IYBI is reporting only one internal equity passing link. I've somehow never noticed this and now that we are doing a lot more competitor analysis, I'm a little concerned given the numbers some of the other sites in the space are getting. I'm not sure I understand it completely and how it's possible we only have one. Any help would be appreciated.
-
Hello!
We do have another bot for our Mozscape Index (Open Site Explorer) which is dotbot
We do find ourselves blocked by a hosting provider at least once a week but mostly as a result of miscommunication between marketers working with clients with multiple web developers. One or two of them may not be aware of a crawler being setup to crawl a site and will see it as malicious behavior, thus causing our bot to be blocked. This is mostly seen from larger companies.
-
Hi David,
I've since spoken to our host at WP Engine and they confirmed they were blocking rogerbot. Is dotbot another crawler from MOZ I need to make an exception for?
Seems INSANE to me that our host would just block your bots for no reason and without notification. Do you run into this often? Any thoughts on why?
-
These pages can be found under the top pages tab in OSE: http://moz.com/researchtools/ose/pages?page=1&site=http%3A%2F%2Fifyoubuildit.com.au
The pages appear to not like our crawler dotbot when trying to request an HTTP response
curl -A "dotbot" http://ifyoubuildit.com.au/branding/
<title>403 Forbidden</title>
<center>
403 Forbidden
</center>
<center>nginx/1.2.9 WPEngine/6.0.7</center>
-
Isn't a 403 a permissions error? It seems impossible to me that some of our high ranking pages are returning this error and we have not noticed before. Where are you seeing this info?
-
Wow. Seems crazy. Would something like that effect our ranking as well?
-
Hello!
It appears the site returns a 403 HTTP response for many of your top pages which contribute to internal links to your home page. Some we are not able to reach at all and others return the 403 response. So you would want to check with your hosting provider and web developer to make sure the chmod permissions are the same which is the most common issues we've seen for these errors.
Check the permission set for this page http://ifyoubuildit.com.au/2014/07/golden-age-cinema-bar/ which we can reach and compare it to one we cannot such as http://ifyoubuildit.com.au/branding
Hope this helps!
-
Okay thanks for the info Erin. I guess my concern is that our site has been around for years. It's not new and it's the one metric that doesn't seem to make any sense.
Are you able to look at our site and tell me if there are any crawl issues?
-
Hi there! Thanks for reaching out. My name is Erin, and I'm on the Moz Help Team. I appologize that it's taken us so long to get back to you, things have been a little hectic around here!
I know this might look a little wonky in OSE, but it's just because of the way we build our index. It's totally possible that you have more internal equity passing links, but that not all of them are in our index yet. Most new sites and links will be indexed by our spiders and available in Mozscape and Open Site Explorer within 60 days, but some take even longer for many of reasons, including the crawl-ability of sites, the amount of inbound links to them, and the depth of pages in subdirectories.
Just so you know, here's how we compile our index:
- We grab the most recent index.
- We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
- We start crawling from the top down until we've crawled 90,000,000,000 pages (which is about 35% the amount in Google's index).
Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index. Sorry!
We update our Mozscape Index every 4 weeks. Crawling the entire Internet to look for links takes 2-3 weeks, but our crawlers are always collecting data. When we need to put the index together, we grab all the data they have collected and start processing which can take up to 3 weeks to determine which of those links are the most important. You can see our most recently updated schedule here: http://moz.com/products/api/updates
Mozscape focuses on a breadth-first approach. Therefore we almost always have content from the homepage of websites, externally linked-to pages, and pages higher up in a site's information hierarchy. However, deep pages that are buried beneath many layers of navigation are sometimes missed and it may be several index updates before we catch all of these.
If our crawlers or data sources are blocked from reaching those URLs, they may not be included in our index (though links that point to those pages will still be available). Finally, the URLs seen by Mozscape must be linked-to by other documents on the web or our index will not include them.
Whew! Sorry for the long answer, I just wanted to be as thorough as possible. I hope this helps, and if you have any other questions, just shoot us an email at help@moz.com.
Cheers!
Erin
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Internal Equity-Passing Links not getting crawled in Moz Open Site Explorer?
Internal Equity-Passing Links not getting crawled in Moz Open Site Explorer. What is the cause of this? We've checked the robots.txt and htaccess file, but so far we can't find anything that would be blocking Moz from crawling the internal links. We manage loads of other clients on this platform and this is the first time we've run into this issue. What else can I check?
Link Explorer | | OozleMedia0 -
Drop in DA and PA and lost a load of links
Hi Mozzers, I just check a site in OSE and while the inbound domains are up the total links have dropped and therefore DA and PA. How can I find which links have gone. I'm assuming a domain passing 1000's of links has either gone down or broken the link - can I use the ISE tool to fine which links have been recently broken? I know there is just discovered for new links. If not can you recommend a tool please. Thanks
Link Explorer | | Bush_JSM0 -
What happened to my links in Open Site Explorer.
Hello, A while ago I had a problem with open site explorer saying that I had a problems with redirects on my home page. There was 302 redirect and it was giving me information from my www domain rather then from non www domain. With your awesome help, this issue is sorted now and when I type in my non www domain, it doesn't give me that weird message anymore. I have a different issue now. For some reason when I enter non www domain it doesn't show links from www version. I presume that this is affecting my site authority for non www version. Does it mean that link juice was not passed on to non www version from www version? Authority dropped by two points, but I'm not sure why. I'm more interested to know why it doesn't show my links from www version on non www version. And is that affecting my sites rankings. Thank you, Regards, Armands
Link Explorer | | A_Fotografy0 -
None of the pages crawled contain an email address or links to a social profile...
I'm trying to reduce the amount of spam flags our website has using the Open Site Explorer tool. Currently, we have 2/17 flags based on: Large Site with Few Links - We found very few sites linking to this site, considering its size No Contact Info - None of the pages crawled contain an email address or links to a social profile The "few links" can be ignored, we're working on this. We don't have a visible email address on the website and we don't particularly want one. We prefer customers to fill out our online form or to call us. Moz say "none of the pages crawled contain an email address OR links to a social profile" - we do have social buttons on every page of the website, but these are official Facebook and Twitter buttons that are rendered with Javascript, so don't actually appear in the page source on load. If we replace these with actual links to our pages using Facebook and Twitter icons, will this flag be removed since Moz are saying "or links to a social profile" - making it sound optional. Thanks!
Link Explorer | | LiamMcArthur0 -
Learn how to use Open Site Explorer's Just-Discovered Links report to build links and manage your brand. Get your Daily SEO Fix!
The Just-Discovered Links report in Open Site Explorer helps you discover recently created links--within an hour of them being published. It's pretty nifty. In today's Daily SEO Fix, Nick shows you how to use the report to view who is linking to you, how they're doing it, and what they are saying so you can capitalize on link opportunities while they're still fresh plus, monitor and participate in the conversations happening about your brand. Watch our tutorial Using Open Site Explorer to Find Fresh Links and Manage Your Brand Online and get ready to dive in! This video is part of The Moz Daily SEO Fix tutorial series--Moz tool tips and tricks in under 2 minutes. To watch all of our videos so far, and to subscribe to future ones, make sure to visit the Daily SEO Fix channel on YouTube.
Link Explorer | | kellyjcoop3 -
Why does Open site explorer only show a fraction of the linked Domains that Google does?
One of my sites ( http://www.georgerossphotography.com ) shows 19 root domains and 155 links on Open Site Explorer. This doesn't pass the sniff test with me and when I look at Google Webmasters they show 120 domains with 1,870 links. I am assuming that Google use it's own data for ranking purposes which makes me question the validity of Open Site Explorer? Am I missing something ? I have been using OSE as my primary tool and now I feel that it has little value. I would appreciate any feedback. George.
Link Explorer | | sirgeorge0 -
Why won't Open Site Explorer detect inbound links from big name websites?
I understand smaller, spammy sites woudn't get indexed, but some of the websites linking to mine have high domain authorities. The website is a parked subdomain.
Link Explorer | | myers3680 -
Open Site Explorer not detecting linking domain on my site
So I was using Open Site Explorer to analyze my domain authority and page authority as can be seen here. My site has been covered by some press and in the article it actually links my site. Some of the articles can be seen on, here, here, and here. Any idea why none of these articles are detected as a linking domain to my site? Is there something that I am doing wrong?
Link Explorer | | herlamba1