Tons of Crappy links in new OSE (Open Site Explorer)
-
I am starting to miss the old OSE. I've found that for a lot of the pages on our site, the new OSE is showing WAY more links and most of them are garbage nonsense links from China, Russia, and the rest of the internet Wild West.
For instance, in the old OSE, this page used to show 9 linking domains:
http://www.uncommongoods.com/gifts/by-recipient/gifts-for-him
It now shows 454 links. Some of the new links (about 5 of them) are legitimate. The other 400+ are garbage. Some are porn sites, most of them don't even open a web page, they just initiate some shady download. I've seen this for other sites as well (like Urban Outfitters) This is making it much harder for me to do backlink analysis on bc I have no clue how many "Normal" links they have. Is anyone else having this problem ? Any way to filter all this crap out ? See attached screenshot of the list of links I'm getting from OSE.
-
Ok thank you. I will email directly.
-
Hey Zack,
Sorry to hear you're still having problems - we've seen an improvement on most sites at this point. Would you want to send me info on the site you're searching and any filters you are using?
If you don't feel comfortable posting that info on this thread, feel free to email me directly: carin@seomoz.org.
Thanks!
Carin
-
Hey Carin,
I just wanted to follow up on this...I'm still seeing these spammy binary files show up as links. Unfortunately it makes OSE quite useless for me in regards to exploring our own backlinks.
What is the status of this problem? Has there been any headway ? Why does our site have problems but most others don't?
Thanks!
-Zack
-
Hey Zack,
Thanks so much for understanding! We are doing everything we can to get the bug resolved. Binary files are the downloadable files you see as links - .pdf, .exe, .img, etc.
I'm really sorry, but we don't have a URL to the old OSE. I saw Steven's response as a workaround - is that possible or are there too many file types to filter out?
Our crawlers that provide the metrics to OSE are always crawling, but will take about a month for our fix to propagate through to all the pages we crawl. Once we have removed these links from our crawlers, then we'll have to process the metrics. This is why it's looking like late September for the fix to show up.
I really appreciate your patience and understanding, we're doing everything we can to fix it!!
Thanks,
Carin
-
Hey Carin-
Thank you so much for this in-depth response. Glad to hear that you guys are aware of it and trying to sort it out. Very interesting info...I'd never hear of "binary" links before but I hope you guys can figure out how to handle these. Seems like a tough task to tackle, just by looking at my CSV it looks like these come in several different forms and they could be hard to identify..I have a few questions:
1. Is there by chance a URL you could give me that points to the old OSE ?
2. How often does OSE crawl? Is it a constant process or are there scheduled crawls?
Thanks!!
-Zack
-
Hey Zack, I saw the ticket you filed was answered by Aaron, but I just wanted to follow up with you as well. We have made some really exciting changes to the crawler, but, unfortunately, there is a pretty obvious bug as well...
The reason for the “questionable” links coming from the Internet Wild West is due to the crawler reaching much deeper into sites where there are more download (i.e. binary) links. The first issue is the crawler is counting a binary file as a link, but the larger issue, is that the crawler doesn’t really know how to handle these types of files. This bug is causing some links to be improperly associated with certain domains. This is probably what you're seeing with all the crazy links from China and Russia which don't actually link to the site you're researching.
There are two steps to addressing this issue: changing how the crawler sees these file types and then fixing how the crawler handles these file types. We have made improvements to our algorithm so that we will be handle the majority of these files correctly, however, this update will need about a month to propagate. The fix for this issue probably won’t be seen for two more updates, meaning late September. Our improvements should catch most of the issues, but there still could be a few cases we haven't addressed. If this happens, don't hesitate to let us know; we love feedback since it helps us improve and make our index even better!
The next step is to fix how our crawlers handle binary file links and prevent them from being improperly associated with certain domains. We are in the process of working through that issue right now. We’re doing everything we can to resolve this bug as we know it is alarming to see these “questionable” links associated with your sites.I hope this helps and thanks so much for being patient :)Thanks,Carin
-
2 ways:
- Get as CSV and spend the time going through it
- Wait it out
-
OK cool good info, hope they fix it soon!! Any good ideas on how you can filter this crap[ out ?
-
Hello Zack,
That is an issue that they are working on, I know this because I already discussed this with one of their help desk people. Here is the page that describes the changes: http://www.seomoz.org/blog/brand-new-open-site-explorer-is-here
In addition to that, here is some additional information I can share with you:
you may see “questionable” links with weird file extensions. This is due to the crawler reaching much deeper into sites where there are more download links. We are looking into fixing this bug as soon as we can so these won’t be counted as links.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I mprove site visibility and keyword ranking for new product site
Hi, Sorry if this is a ridiculous post as I am really new to SEO, but I haven't had this problem with other sites! We had a website www.r-dna.co.uk that was never promoted or used very much as it was early days in the product lifecycle. The product (is called R-DNA or Remote Data Network Analysis) is now live so we re-branded and re-launched the site - it has now been live since the beginning of September but we still only have 0.35% visibility and very little ranking in our keywords. We are also using Google Adwords to try and generate business and have registered with numerous online business directories. I have been blogging to update content, tweeting and updating our facebook page, but we still aren't getting the traffic or visibility increases that we have experienced with our other sites. The MOZ site crawl shows 5 medium priority issues (duplicate title page & missing meta description tag), but no major issues. I know its probably fairly early days for a "new" site, but wondered if anyone could advise if there is anything wrong which would explain our lack of visibility.
Moz Pro | | sharon.bathurst0 -
Site explorer Issue
Hello, I'm looking to see in the Site Explorer the links coming from directories such as BOW, yahoo etc. I'm listed there from almost 1 year and these links are not listed, the same with my competitors. I'm missing something? Thank you Claudio
Moz Pro | | SharewarePros0 -
"Too many on page links" phantom penalty? What about big sites?
So I am consistently over the recommended "100 links" rule on our site's pages because of our extensive navigation and plentiful footer links (somewhere around 300 links per page). I know that there is no official penalty for this but rather that it affects the "link juice" of each link on there. I guess my question is more about how places like Zappos and Amazon get away with this? They have WAY over 100 links per page... in fact I think that Zappos footer is 100+ links alone. This overage doesn't seem to affect their domain rankings and authority so why does SEO moz place so much emphasis on this error?
Moz Pro | | kida12meyer0 -
How do I collect all links through the SEOMoz API?
How do I collect all links through the SEOMoz API? Yes, I know how to pull url metrics info but want to pull the inbound links only via php script w/out having to go to the OSE page to pull. any suggestions would be great!
Moz Pro | | destdigi0 -
Old data in OSE?
Hello, I know the Mozscape was recently updated, but it's been a week and my site data ( DA, number of backlinks) in Open Site Explorer is still the same. The seomoz toolbar and my pro tools show my DA has increased ( I don't know of another way to verify my backlink count, so I'm not sure on that front). Any idea of the reason behind the disparity? Thanks.
Moz Pro | | richje0 -
Where is the best place to add links on my site?
If I'd like to put links to other sites in my site, is it better to have a page named "Our Helpful Links" etc. instead of just adding them to the bottom of an existing page like I've seen on some sites? I'm asking because I'm wanting to make Google as happy as possible and still add them. Just in case it helps to look at the site yourself to give advise its; http://www.allstatetransmission.net If you see anything else there that I should work on feel free to be hard on it, I value any criticism. Thanks, Jeff
Moz Pro | | allstatetransmission0 -
Links not appearing on Open Site Explorer
My site gained several new inbound links during December and only two of them are not all showing up on the latest Linkscape update. It seems to be the links that were created at the end of the month which are showing up, whereas a handful at the beginning of the month are nowhere to be seen. All the linking pages have been indexed by Google the links are do-follow, and one of the sites in particular is not obsure and has a DA in the 90's. I appreciate the Linkscape doesn't index everything, but I would have thought that more tof the results of my efforts would have shown up in OSE. I'd be really grateful if anyone could explain this to me please. Thanks Ben
Moz Pro | | atticus70 -
Competitive .edu Research via Open Site Explorer
I was using open site explorer and trying to figure out how my competitors are getting so many .edu links. Now I won't mention any names here but I am trying to figure out why almost all their links point to downloads of documents. Here are a few of the examples of sites I keep finding... www.cs.uoregon.edu/research/paracomp/tau/monitor/sheehan_examples/tim_context_xmpl?wiki=TailleMaximaleDesFichiersSurUnePartition/orphan www.atmos.albany.edu/facstaff/mathias/video/090926_wima_klimawandel.ivr?url_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.jtitle=American Journal Clinical Nutrition&rft.atitle=Problems with red meat in the WCRF2&rft.volume=89&rft.spage=1274&rft.epage=9&rft.date=2009&rft.aulast=Truswell&rft.aufirst=AS&rfr_id=info%3Asid%2Fwiley.com%3AOnlineLibrary faculty.unlv.edu/jensen/CEE_468/GISTutorialWorkbook/tutorial11/LandUsePgh.lpk?arrow=nws&read=24922 <colgroup><col width="3111"></colgroup>
Moz Pro | | MichealGooden0