Changing Links to Spans with Robots.txt Blocked Redirects using Linkify/jQuery
-
Hi,
I was recently penalized most likely because Google started following javascript links to bad neighborhoods that were not no-followed. The first thing I did was remove the Linkify plugin from my site so that all those links would disappear, but now I think I have a solution that works with Linkify without creating crawlable links. I did the following:
-
I blocked access to the Linkify scripts using robots.txt so that Google won't execute the scripts that create the links. This has worked for me in the past with banner ads linking to other sites of mine. At least it appears to work because those sites did not get links from pages running those banners in search console.
-
I created a /redirect/ directory that redirects all offsite URLs. I put a robots.txt block on this directory.
-
I configured the Linkify plugin to parse URLs into span elements instead of a elements and add no follow attributes. They still have an href attribute, but the URLs in the href now point to the redirect directory and the span onclick event redirects the user.
I have implemented this solution on another site of mine and I am hoping this will make it impossible for Google to categorize my pages as liking to any neighborhoods good or bad. Most of the content is UGC, so this should discourage link spam while giving users clickable URLs and still letting people post complaints about people that have profiles on adult websites. Here is a page where the solution has been implemented https://cyberbullyingreport.com/bully/predators-watch-owner-scott-breitenstein-of-dayton-ohio-5463.aspx, the Linkify plugin can be found at https://soapbox.github.io/linkifyjs/, and the custom jQuery is as follows:
jQuery(document).ready(function ($) { 2 $('p').linkify({ tagName: 'span', attributes: { rel: 'nofollow' }, formatHref: function (href) { href = 'https://cyberbullyingreport.com/redirect/?url=' + href; return href; }, events:{ click: function (e) { var href = $(this).attr('href'); window.location.href = href; } } }); 3 });
-
-
I know I was penalized because search console recorded an overnight drop in impressions/clicks of 75%. Never was a problem before. I didn't need to run Screaming Frog or anything to find links that explain the problem. I just browsed some of the user generated content and then once I realized that Google was crawling javascript generated links it all made sense.
Some of the sites people were liking to included: CheaterLand.com, PredatorsWatch.com, DirtyHomeWreckers.com, EscortBabylon.net, SugarDaddyforMe.com, GFEMonkey.com, EscortBabylon.com, CityXGuide.com, and AdultLook.com.
I think as long as Google respects the robots.txt directives I should be just fine. The redirect page itself is blocked by robots.txt so even if Google finds one of those URLs (ex: https://cyberbullyingreport.com/redirect/?url=https://moz.com/community/q/changing-links-to-spans-with-robots-txt-blocked-redirects-using-linkify-jquery) it shouldn't even follow the 302 since the robots.txt file (https://cyberbullyingreport.com/robots.txt) disallows that directory.
-
301 or 302 redirecting well if you’re going to do one you should do 302 but it’s not going to help you much unless you’re going to send the URL to a 410 when something is no followed it is the same thing as what robots.txt will do.
https://support.google.com/webmasters/forum/AAAA2Jdx3sUEbHp0yjgT6c?hl=sv
do you have a report from Google knowing that you have been penalized?
is there anyway you could run Screaming Frog and show some of these URLs that you’re talking about?
respectfully,
Tom
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO Links in Footer?
Hi, One of my clients uses a pretty powerful SEO tool, won't mention the name. They now have a "link equity" tool, which they are using on a lot of their client's sites, which include tons of fortune 500 companies. It involves add footer links to your site that change based on the content of the page they are on. The machine learning tries to figure out the most related pages and links to them with the heading tag of that page as the anchor text. Initially this sounds very spammy to me. But then, it seems a lot like "related products" tools that many companies use. The goal for this tool is to build up internal linking, especially for deeper pages on their site. They have over 10,000 currently. What are everyone's thoughts on this strategy?
White Hat / Black Hat SEO | | vetofunk2 -
How many links can you have on sitemap.html
we have a lot of pages that we want to create crawlable paths to. How many links are able to be crawled on 1 page for sitemap.html
White Hat / Black Hat SEO | | imjonny0 -
How does Google handle product detail page links hiden in a <noscript>tag?</noscript>
Hello, During my research of our website I uncovered that our visible links to our product detail pages (PDP) from grid/list view category-nav/search pages are <nofollowed>and being sent through a click tracking redirect with the (PDP) appended as a URL query string. But included with each PDP link is a <noscript>tag containing the actual PDP link. When I confronted our 3rd party e-commerce category-nav/search provider about this approach here is the response I recieved:</p> <p style="padding-left: 30px;">The purpose of these links is to firstly allow us to reliably log the click and then secondly redirect the visitor to the target PDP.<br /> In addition to the visible links there is also an "invisible link" inside the no script tag. The noscript tag prevents showing of the a tag by normal browsers but is found and executed by bots during crawling of the page.<br /> Here a link to a blog post where an SEO proved this year that the noscript tag is not ignored by bots: <a href="http://www.theseotailor.com.au/blog/hiding-keywords-noscript-seo-experiment/" target="_blank">http://www.theseotailor.com.au/blog/hiding-keywords-noscript-seo-experiment/<br /> </a> <br /> So the visible links are not obfuscating the PDP URL they have it encoded as it otherwise cannot be passed along as a URL query string. The plain PDP URL is part of the noscript tag ensuring discover-ability of PDPs by bots.</p> <p>Does anyone have anything in addition to this one blog post, to substantiate the claim that hiding our links in a <noscript> tag are in fact within the SEO Best Practice standards set by Google, Bing, etc...? </p> <p>Do you think that this method skirts the fine line of grey hat tactics? Will google/bing eventually penalize us for this?</p> <p>Does anyone have a better suggestion on how our 3rd party provider could track those clicks without using a URL redirect & hiding the actual PDP link?</p> <p>All insights are welcome...Thanks!</p> <p>Jordan K.</p></noscript></nofollowed>
White Hat / Black Hat SEO | | eImprovement-SEO0 -
Being Link Attacked - Should I worry?
Hey, Hope everyone is well. Just a quick question. I hope to get an answer from Google officially (I've asked in their webmaster forums area) but any experience or opinions from the community here would be great. I noticed recently that our site started to get thousands of links from comments in random blogs from all across the web. This is nothing to do with us as we don't "build links". I can only assume it is a competitor trying to get our site hit by the algorithm for a particular search term, as all the anchor text (I estimate about 1,800 links with this anchor text) point to one page on our site that is ranking for that term. I recently removed the website from webmaster tools and re added, due to an unrelated issue about the a video rich snippet not updating, and all the links have just popped up today on there. Is this something I need to worry about? and should I start collecting all these domains and using the disavow tool to block the whole domain of these sites with the comments (some of them seem like genuine sites). There seem to be new ones everyday and it looks to be an ongoing attack as well. Thanks in advance!
White Hat / Black Hat SEO | | JonathanRolande0 -
Removing Poison Links w/o Disavow
Okay so I've been working at resolving former black-hat SEO tactics for this domain for many many months. Finally our main keyword is falling down the rankings like crazy no matter how many relevant, quality links I bring to the domain. So I'm ready to take action today. There is one inner-page which is titled exactly as the keyword we are trying to match. Let's call it "inner-page.html" This page has nothing but poison links with exact match anchor phrases pointing at it. The good links I've built are all pointed at the domain itself. So what I want to do is change the url of this page and let all of the current poison links 404. I don't trust the disavow tool and feel like this will be a better option. So I'm going to change the page's url to "inner_page.html" or in otherwords, simply changed to an underscore instead of a hyphen. How effective do you think this will be as far as 404ing the bad links and does anybody out there have experience using this method? And of course, as always, I'll keep you all posted on what happens with this. Should be an interesting experiment at least. One thing I'm worried about is the traffic sources. We seem to have a ton of direct traffic coming to that page. I don't really understand where or why this is taking place... Anybody have any insight into direct traffic sources to inner-pages? There's no reason for current clients to visit and potentials shouldn't be returning so often... I don't know what the deal is there but "direct" is like our number 2 or 3 traffic source. Am I shooting myself in the foot here? Here we go!
White Hat / Black Hat SEO | | jesse-landry0 -
Has anyone used this? www.linkdetox.com/
Has anyone used this? www.linkdetox.com/ Any opinions about it?
White Hat / Black Hat SEO | | Llanero0 -
Index page de-indexed / banned ?
Yesterday google removed our index page from the results. Today they also removed language subdomains (fr.domain.com).. Index page, subdomains are not indexed anymore. Any suggestions? -- No messages in GWT. No malware. Backlink diversification was started in May. Never penguilized or pandalized. Last week had the record of all times of daily UV. Other pages still indexed and driving traffic, left around 40% of total. Never used any black SEO tool. 95% of backlinks are related; sidebar, footer links No changes made of index page for couple months.
White Hat / Black Hat SEO | | bele0 -
Too many links with the same Anchor-text?
My first question at SeoMoz: Recently my gambling site has been experimenting a subtle yo-yo effect for our most sought-after keyword. A month ago we legitimately added a PR-6 inbound link with that keyword (tragamonedas) from an institutional site of our own development. We are worried that google might have regarded that move as an illegitimate link acquisition, since those apparent troubles with our keyword appear to have started right after that link was processed. Is it too late to change the anchor text, in case that action might deliver positive results? Also, we might have focused too much on the very same keyword in our link building campaign. Can a constant repetition of the same anchor harm our indexing reputation? Thank you in advance and good SEO luck, Andi.
White Hat / Black Hat SEO | | castano0