Only one internal Equity Passing Link
-
Our web site IYBI is reporting only one internal equity passing link. I've somehow never noticed this and now that we are doing a lot more competitor analysis, I'm a little concerned given the numbers some of the other sites in the space are getting. I'm not sure I understand it completely and how it's possible we only have one. Any help would be appreciated.
-
Hello!
We do have another bot for our Mozscape Index (Open Site Explorer) which is dotbot
We do find ourselves blocked by a hosting provider at least once a week but mostly as a result of miscommunication between marketers working with clients with multiple web developers. One or two of them may not be aware of a crawler being setup to crawl a site and will see it as malicious behavior, thus causing our bot to be blocked. This is mostly seen from larger companies.
-
Hi David,
I've since spoken to our host at WP Engine and they confirmed they were blocking rogerbot. Is dotbot another crawler from MOZ I need to make an exception for?
Seems INSANE to me that our host would just block your bots for no reason and without notification. Do you run into this often? Any thoughts on why?
-
These pages can be found under the top pages tab in OSE: http://moz.com/researchtools/ose/pages?page=1&site=http%3A%2F%2Fifyoubuildit.com.au
The pages appear to not like our crawler dotbot when trying to request an HTTP response
curl -A "dotbot" http://ifyoubuildit.com.au/branding/
<title>403 Forbidden</title>
<center>
403 Forbidden
</center>
<center>nginx/1.2.9 WPEngine/6.0.7</center>
-
Isn't a 403 a permissions error? It seems impossible to me that some of our high ranking pages are returning this error and we have not noticed before. Where are you seeing this info?
-
Wow. Seems crazy. Would something like that effect our ranking as well?
-
Hello!
It appears the site returns a 403 HTTP response for many of your top pages which contribute to internal links to your home page. Some we are not able to reach at all and others return the 403 response. So you would want to check with your hosting provider and web developer to make sure the chmod permissions are the same which is the most common issues we've seen for these errors.
Check the permission set for this page http://ifyoubuildit.com.au/2014/07/golden-age-cinema-bar/ which we can reach and compare it to one we cannot such as http://ifyoubuildit.com.au/branding
Hope this helps!
-
Okay thanks for the info Erin. I guess my concern is that our site has been around for years. It's not new and it's the one metric that doesn't seem to make any sense.
Are you able to look at our site and tell me if there are any crawl issues?
-
Hi there! Thanks for reaching out. My name is Erin, and I'm on the Moz Help Team. I appologize that it's taken us so long to get back to you, things have been a little hectic around here!
I know this might look a little wonky in OSE, but it's just because of the way we build our index. It's totally possible that you have more internal equity passing links, but that not all of them are in our index yet. Most new sites and links will be indexed by our spiders and available in Mozscape and Open Site Explorer within 60 days, but some take even longer for many of reasons, including the crawl-ability of sites, the amount of inbound links to them, and the depth of pages in subdirectories.
Just so you know, here's how we compile our index:
- We grab the most recent index.
- We take the top 10 billion URLs with the highest MozRank (with a fixed limit on some of the larger domains).
- We start crawling from the top down until we've crawled 90,000,000,000 pages (which is about 35% the amount in Google's index).
Therefore, if the site is not linked to by one of these seed URLs (or one of the URLs linked to by them in the next update) then it won't show up in our index. Sorry!
We update our Mozscape Index every 4 weeks. Crawling the entire Internet to look for links takes 2-3 weeks, but our crawlers are always collecting data. When we need to put the index together, we grab all the data they have collected and start processing which can take up to 3 weeks to determine which of those links are the most important. You can see our most recently updated schedule here: http://moz.com/products/api/updates
Mozscape focuses on a breadth-first approach. Therefore we almost always have content from the homepage of websites, externally linked-to pages, and pages higher up in a site's information hierarchy. However, deep pages that are buried beneath many layers of navigation are sometimes missed and it may be several index updates before we catch all of these.
If our crawlers or data sources are blocked from reaching those URLs, they may not be included in our index (though links that point to those pages will still be available). Finally, the URLs seen by Mozscape must be linked-to by other documents on the web or our index will not include them.
Whew! Sorry for the long answer, I just wanted to be as thorough as possible. I hope this helps, and if you have any other questions, just shoot us an email at help@moz.com.
Cheers!
Erin
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Internal Links is reporting "0" value
Open Site Explorer is reporting "0" value under "Total Internal Links" as well as "Internal Links" metrics for my website. I have a good interlinking of pages throughout the site.
Link Explorer | | anupjain
Is there an issue with OSE (Open Site Explorer) crawler ? OR There are some crawling problems with our Website's structure ?
Can anybody please give an idea or recommendations on this ?
Domain: https://www.traveltoindia.org/0 -
The New Link Explorer (which will replace Open Site Explorer) is Now in Beta
Howdy Moz friends, Today, the Moz team is making a new tool -- Link Explorer -- available in private beta for Moz Pro subscribers (including those taking a free trial). The tool is still a little ways out from public launch, but we wanted to get your feedback to help make it the best product it can be. What's Link Explorer? In essence, it's a replacement for Open Site Explorer (Moz's tool for link discovery, competitive analysis, and link building) that addresses many of its most pernicious challenges, such as: Daily updates - no longer will you need to wait a month or two to see a new DA score or the links you built last week. Link Explorer updates every 24 hours with all the new links we've discovered that day, and gives a new DA score each night. A MUCH bigger index - OSE has always been known for having quality links, but quantity has been an issue. No longer. The new tool's link index is more than 20 times larger than Open Site Explorer's, covering trillions of links across hundreds of billions of pages, while maintaining a focus on high quality domains and pages. Additional functionality - new graphs (like link growth over time), new charts (like gained+lost links), new filters and sorts, and some new kinds of data coming soon. Improved metrics - Domain Authority and Page Authority have both been upgraded to have better correlations with Google's rankings (and they now update every 24 hours) Insanely fast - page load times on the new tool are almost as fast as Google's results 🙂 Less time waiting means more time to dig into results Link Tracking Lists - check a box next to any links of interest and you can build lists in the tool to track them over time, see whether/when they link, prioritize your outreach efforts, and (in the future) get aggregated data and alerts about those links There's much more to come, but we'd love for you to check out Link Explorer, find bugs, report things you love (and don't), and help us make it the best possible product for you and your teams. You can leave feedback here in this Q+A thread, email help@moz.com, or send feedback through the feedback form in Link Explorer.
Link Explorer | | randfish18 -
Is this site linking to us?
Hi, Moz OSE does not recognize this site as linking to our site, but I feel like it has to be. Here is the code and the link to the site: http://www.avvo.com/attorneys/34655-fl-stacy-kemp-1843513.html#contact [https://kemprugegreen.com/location/tampa/tampa-car-auto-accident-attorney/](<a class=)" target="_blank">https://kemprugegreen.com/location/tampa/tampa-car-auto-accident-attorney/ So are they linking to us? If so, how come OSE cant find it? It lists it for our competitors. Thanks, Ruben
Link Explorer | | KempRugeLawGroup0 -
Not existing domains linking to my website (spam)
Hello, When i run my platform www.taobao.nl through https://moz.com/researchtools/ose/ i see a lot of nasty adult domains that link to my platform. These give a very negative (spam)score to my platform. I already disavowed quite a few via Webmastertools, but they keep coming. When i check the names in the Whois, they don't even seem to exist! (anymore) What could cause this and how can i end it?? Thanks for your help! Sander
Link Explorer | | benhond1 -
Identifying Recently Added Links?
Is there a way to see specifically which links were added in my campaign? I noticed that this week we have 133 new links that were added to out website, yet we lost 1 link on the root level. So I am a little confused where that is coming from. I wanted to see if I could see where. When I look at the recent inbound links in the OpenWeb explorer there isn't anything listed. Thanks!
Link Explorer | | HashtagHustler0 -
How can I use Open Site Explorer to Identify why my own URL's are appearing as inbound links
I'm a new member of Moz, and am trying to understand why I am seeing my sites URL's in my Inbound Link results in Open Site Explorer. There are quite a few, and they are all product category URL's that are generated by my stores database. This is not happening with .html URL's. Any thoughts?
Link Explorer | | Artfx0 -
Does Open Site Explorer recognise nofollow links?
In the backlink profile for my site on OSE, I'm seeing a lot of links that are paid advetorials and I know go through DoubleClick. I was informed that links going through Doubleclick automatically become NoFollow links. If the above is true, should I ignore these links? If they are NoFollow, they don't really contribute to my backlink profile? THANKS!
Link Explorer | | whis0 -
External Followed Links report
I would like to know, how do I get a report with the TOTAL amount of my external followed links.When I look on "Link Metric" it shows me: External Followed Links 2,951 But when I looked on Inbound links (only follow / only external), it shows me only 645. Why is that and how do I get the total from 2.951? Thanks
Link Explorer | | AutoEurope0