Gotcha. Then Dana's suggestions will be good. The challenge you'll probably encounter is one of matching cost for an audit against the business value. The companies recommended by Moz tend to be at the higher end of cost, because they're large companies with plenty of experience and expertise.

Paul

You need to be aware, Jonathan, that there is absolutely nothing about a robots.txt disallow that will help remove a URL from the search engine indexes. Robots is a crawling directive, NOT an indexing directive. In fact, in most cases, blocking URLs in robots.txt will actually cause them to remain in the index even longer.

I'm assuming you have cleaned up the site so the actual spam URLs no longer resolve. Those URLs should now result in a 404 error page. You must confirm they are actually returning the correct 404 code in the headers. As long as this is the case, it is a matter of waiting while the search engines crawl the spam URLs often enough to recognise they are really gone and remove them from the index. The problem with adding them to the robots.txt is that is actually telling the search engines NOT to crawl them, so they are unlikely to discover that they lead to 404s, hence they may remain in the index even longer.

Unfortunately you can't use a no-index tag on the offending pages, because the pages should no longer exist on the site. I don't think even a careful implementation of a X-Robots noindex directive in htaccess would work, because the URLs should be resulting in a 404.

Make certain the problem URLs return a clean 404, use the Google Search Console Remove URLs tool for as many of them as you can (for example you can request removal for entire directories, if the spam happened to be built that way), and then be patient for the rest. But do NOT block them in robots.txt - you'll just prolong the agony and waste your time.

Hope that all makes sense?

Paul

No-following links has ABSOLUTELY ZERO EFFECT on preserving "link juice" of a page, Rich. This used to be the case six years ago when no-follow for links was first introduced, but it was being abused so badly that search engines changed this behaviour. (This used to be referred to as PageRank sculpting)

Further to Andy's and Dmytro's comments - Google is clear there are only three circumstances when no-follow should be used:

1. you have a commercial relationship with the page you're linking too (paid links, but also many guest post scenarios for example)
2. you didn't create the link and therefore can't trust it (e.g. user comments or other user generated content)
3. you are linking to an unreliable site (to demonstrate a bad example,for instance)
4. (and a bonus fourth) links to administrative-type pages that wouldn't be of any use to a search visitor like a privacy/terms of service or login page).

There's also been considerable discussion that Google in particular considers no-following of all external links a sign of unnatural manipulation that could damage page authority.

So... conceptually a good idea at one time, but no longer valid and potentially harmful.

Hope that helps?

Paul

Yup - Google still says content that can only be seen after a user interaction is given less importance. Kinda stupid, given that things like tabs/accordians are a major usability enhancement, but that's still where we are.

P.

The example page does have a noindex tag in place, but it's not formatted correctly, so it's being ignored. Very subtle issue, but your tag is using "smart quotes" around the elements instead of the plain quotation marks that are required for code. If you look very carefully at the page source code, you'll see that they are quotation marks like you'd see in a Word document; the ones at the beginning of robots and noindex curl a different way than the ones at the end.) This usually occurs when the content was written in a word processor instead of a plain-text editor.

Because the tag's not formatted correctly, it's ignored by both the crawling tools and the search engines.

In addition, the site also has all pages blocked from crawling by the sitewide robots.txt file. This and noindex are conflicting instructions to search engines.

If a page is blocked in robots.txt, then the search engine will not crawl the page and so is not able to discover the noindex tag, even if it were formatted correctly. Therefore if the search engine becomes aware of the page in any other way than straight crawling (and there are a number of ways this can happen), then the page will still get indexed.

If it's a dev site, the proper way to keep it from being indexed is to either noindex all pages, or to put the site behind a password so the search engines and public visitors can't access it. If using noindex, the site must not be blocked with a robots.txt directive.

Does that all make sense?

Paul

In addition to Vahe's suggestion, it's possible the terms being used refer to what is also frequently referred to as "the long tail" of search.

Tier 1 or "head" terms are search terms with both extremely high search volume and very high competition Tier 2 or "body" terms have lower search volume and competition, and tier 3 or "long tail" terms are those which are so specific they are searched much less frequently, but also there are considerably fewer sites competing for their traffic.

Example:
tier 1 head term - "SEO"
tier 2 body term - "ecommerce SEO specialist"
tier 3 long tail term - "SEO specialist for WordPress in Seattle"

Note it's not just the # of words in the term that determine it's tier, though usually longer terms will be located farther down. It's more the search volume and amount of competition that determines the classification.

There's lots of info on the long tail of search here at SEOMoz and the web in general if you want to get a deeper understanding of it's value.

Paul

You'll absolutely want to get the blog moved to the subdirectory. Tell your developers they must use 301 redirects to point the traffic from the old subdomain pages to the new location. As long as the blog's site structure and URLs remain the same, that will be a pretty straightforward redirect to write.

Once they've done the redirect, test by going to a bunch of the old page addresses and making sure you end on the new pages.

Also use a header-checking tool on the new pages to confirm that it's actually showing that 301 redirect was used (not a 302, for example)

The sooner you do this the better, so that new incoming links will be pointing directly to the new location instead of being redirectd through the old location.

if you want best bang for the buck, check for the strongest links that currently point to the old site and see if you can get their webmasters to update them to point to the new URLs. Even pages that are redirected don't pass 100% of their authority through the 301 to the new page, and there is evidence to indicate that the 301s pass less authority as they age.

All good reasons to get the move done as soon as practical.

Good luck!

Paul

P.S. If your blog has a name that people may begin to recognize & remember  (ie that is different from the site name) strongly consider naming the folder /the-blog-name instead of just /blog. Every bit of distinctiveness helps!

As Brian says - depends how the JavaScript is written whether Googlebot can parse the link.

Bigger issue though is that the concept you're talking about is exactly the kind of thing that the Penguin Update was designed to devalue/penalize.

Using this method (assuming the links get parsed) is going top create a glaringly unnatural link profile that will likely draw the unwanted attention of the spam team.

Paul

The citation finder tool at Whitespark (mentioned in one of my other comments) can also be really useful, Brett. One way you can use it is to check out the citations your local competitors have, so you can recognise and go after relevant citation sites that you may not have thought of.

www.whitespark.ca/local-citation-finder/

Well worth a paid account for a couple of months at least. (No affiliation - just a great tool. And Canadian Note that as a Pro member here, there's even a perk for that: www.seomoz.org/pro-perks.

Paul

Mememax, you're talking about a completely different situation.

Screaming Frog/Xenu are only capable of broken links that your own site links to.

The SEOMoz and WMT crawls are finding missing pages that other sites are linking to on your site. I.e. broken incoming links as opposed to broken internal links.

While both are good to check, if you aren't watching for and fixing your 404s that other sites are linking to, you are potentially throwing away hugely valuable incoming link juice. (Hint: it's even more valuable juice than what is passing from your internal links)

A simple example of this occurs when you just move a page to a new URL, and update your own site's navigation. A Screaming Frog crawl would show no 404 as a result (because you updated your own site's link), but all the incoming links to that moved page could now be 404ing. It's entirely possible that other sites were linking to that page. If you don't watch your broken incoming links reports either in SEOMoz to WMT, you won't know that those other sites are still trying to link. And then you'd be throwing away that incoming link equity.

Makes sense?

Paul

P.S, Of course, you can also monitor your 404s in Google Analytics, but that only shows you broken incoming links that are getting traffic. The other tools mentioned will show you nearly all the broken incoming links even if they're not currently driving traffic. By fixing them, you'll rescue the ranking value and also be ready if they later start to drive actual referral traffic.

To be clear, here, Guy, and to flesh out Keri's response, because as a former pro photographer and writer this is a pet peeve of mine...

NOTHING published on the Internet is public domain unless the creator specifically states in writing that it is, and you can prove that he/she did so. Which is what your email to the company seeks to do, so good move.

The simple act of publishing content where it is publicly accessible in no way can be interpreted as allowing anyone else to use the content/granting copyright. If the creator chooses to grant lenient republication rights such a Creative Commons licenses, the material can be used under those license restrictions, but even that is not "public domain". There is a specific license granted and you must adhere strictly to the terms of that license.

This applies to all content (written, video, photo, audio, music etc) and to re-purposing and derivative works. The only practical exception for regular users is "Fair Use" which amounts essentially to quoting very small parts of the original - in order to use it in a review, for example. (Other parts of Fair Use include things like use in textbooks and by libraries)

Bottom line: if a website owner didn't create it from scratch, they have no right to republish/use it without written permission from the creator./copyright holder. And the fact lots of people on the web abuse this law won't protect them if they get caught.

I'm not a lawyer. Don't even play one on the Internets. But anyone who thinks the above doesn't apply to them is asking for a world of hurt. Not to mention ripping off the original creator. If you think you're exempt, better have a good lawyer's proof to back it up.

/rant

Paul

P.S. Some works were copyrighted so long ago (and copyright not renewed) that they have fallen out of copyright protection and are actually in the public domain. But we're talking over 100 years and you'll want to have proof before using.

Yup, that's an RSS feed, moregood. By default, WordPress creates a number of specialised feeds in addition to the basic site feed. These include feeds for Categories, Post Comments, Tags, and Authors.

I'm not clear from your question if this feed lives on your own site or someone elses? If it's on someone else's site, its a perfectly legitimate backlink. Some sites choose to allow their feeds to be indexed by search engines to aid in indexing of their pages. There's no reason or need to remove these links from your backlinks unless they're from a spammy site. This link is just showing that one of the posts in that category feed has a link to your site. You'll probably also have a backlink showing from a regular page on that site as well.

Does that answer your question? Or is that link coming from your won site and you're wondering how to remove those special RSS feeds from your own site?

Paul

Paul

So much this ^^

This is a question for which the only answer is "test your own visitors".

Something else to keep in mind - so much of getting visitors to complete longer forms is about setting expectations and demonstrating WIIFM (what's in it for me) up front.

"A minute and half of your time for these questions helps us give you more useful customised recommendations. We don't want to waste your time." Can go an awful long way to getting buy-in from visitors to complete the form. You've told them why giving the info will help them, and told them how much time is required. So even if they see a multi-step form, they won't get worried it's going to drag on.

Something else to think about.

Paul