Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
I'd second Erwan's questions - did other signals confirm the Toolbar PR update? Have you lost ranking, search traffic, etc.
Not only is TBPR notoriously unreliable, but this could reflect something that happened 3-6+ months ago, and maybe it's reversed by now. It's very, very hard to pin down a timeline from TBPR, and if you were penalized, a timeline is critical.
Could you explain that second paragraph? Why do you want to buy a domain that's not only penalized, but whose links are entirely coming from a site you already own? I think I'm misunderstanding the situation.
I agree with Kurt - in lieu of de-listing or redirects, rel=canonical is about your only option. It's possible it won't be enough, but it's the best you've got by a long shot, given the restrictions.
I haven't seen all the numbers, but I know people at major newspapers using cross-domain canonical, and they'd drop it in a heartbeat if it didn't pass the majority of link equity.
I think the domain move case is more compelling, because now you've got a completely new domain that you can show ranking in place of the old, stronger domain, without redirects in place. At that point, it's unlikely just a fluke.
Ah, I understand what you're saying. I think part of it may be that many of the errors are more technical in nature - they're severe, but not generally "black-hat" or "gray-hat" kind of things. Massive dupe content can get you hit with Panda, but people debate whether that's a Capital-P "penalty" (which some reserve mostly for manual penalties). Unfortunately, it is a bit of a loaded word.
Of course, all the copy is changing in Moz Analytics, so this may be a moot point 
That actually makes much more sense than the way I've had people try to explain it to me I didn't realize a Googler had actually condoned it (although sometimes I find Maile's messages a bit mixed).
So, I can tell you from conversations with SEOs that some have used rel=canonical successfully to pass link-juice. In some cases, I even know people who use it to move sites, and then 301 later, and claim success with this method. Unfortunately, almost none of those case studies are published.
Generally speaking, I still don't think it's a great way to move a resource, and tend toward 301s for that purpose, but all the data I've seen suggests that rel=canonical tends to consolidate link juice. There are exceptions, of course, such as when Google doesn't honor the tag (they don't see it as a duplicate, for example, and think you're trying to game the system), but that's true of 301s as well.
Rand did a Whiteboard Friday a couple of years ago talking about link-equity and cross-domain canonical:
http://moz.com/blog/cross-domain-canonical-the-new-301-whiteboard-friday
I know he's actually a big believer that rel=canonical passes link equity, as or more strongly in some cases than 301-redirects (again, it's pretty situational).
Sorry, I'm confused - we do consider it very serious, and "error" is our top level of priority.
Honestly, I wouldn't use base href's to fix 404s in links - I'd fix the links. It depends on the scope, I guess. If the pages are dynamic or we're talking about a few dozen, fixing the links should be pretty easy. If you have thousands of static pages, then that's a bit different, but then you'd have to edit the base tag across all those pages, anyway. I'm having a hard time envisioning a scenario where fixing the links is much more difficult than putting in the base tags, but it's hard to give specific advice without the details.
This is where things get a bit dicey - I'm not 100% sure that won't remove the main page, too (and how Google handles the trailing "/"). You might need a "/*" wild-card in the Robots.txt. Frankly, I'd ease into it with just one directory. These things never seem to work quite the way in practice that we all say they should in theory.
I don't see it used nearly as much as it used to be, and Google's not always very consistent about interpreting it from an SEO standpoint. So, for example, while <base...>works fine for things like relative image URLs, stylesheets, etc., I wouldn't rely on it for things like the canonical tag (I'd stick to absolute URLs there).</base...>
It really depends on what you're trying to accomplish, though. What are your goals, or what problem are you trying to solve? Are we talking about a site with dozens of pages or thousands?
Oh, sorry - yeah, this is why these questions can be dangerous in the scope of Q&A. If some of the pages in that virtual folder are main nav pages/links, then it's definitely going to look weird to block them (it's a mixed signal, at best). I'm not sure I fully understand the site structure, but my gut reaction is to leave those indexed. The wild-cards should work - the other option would be to give them each their own shorter URLs and not put them in the "/design-your-own-jewelry" folder, but that can be a ton of work, depending on how your site is built (plus, you'd have to 301-redirect the old URLs, which opens up a whole new mess).
I've had a lot of issues where, if pages were already indexed, Robots.txt did a poor job of removing them. Absolutely agree on the crawl budget issue and it's a whole lot easier to remove a folder in Robots.txt, but I've just had a bunch of odd problems with Robots.txt at large scale. If I actually had to do it on my own site, I'd probably:
(1) META NOINDEX the pages
(2) Monitor removal
(3) Once removal was progressing well (80%+), then add to Robots.txt
I understand your concern, but if you don't think that contacting the site to have the links removed is a viable option, then disavow really is your only recourse. The only way to find out if Google will accept that as enough is to try it and see. I wouldn't obsess over this one domain, but when you file for reconsideration, it will be important to show progress in other areas. In other words, maybe you can't remove these 14K links (and will just disavow the domain), but make sure you remove some other, bad links, and show an effort somewhere.
Beyond cooperation from the webmaster, your only other recourse is legal, and that's just not going to be worth the time, money, and grief, in 99% of cases. Plus, if a past SEO company did this, you don't even have much legal recourse - the 3rd-party site didn't do anything wrong.
I have to second Matt - both and neither are probably right. This is an extremely difficult thing to measure, especially at large scale, and only Google has the numbers. Even purely organic CTR studies (not accounting for paid search) show numbers all over the place. The one consistent aspect is that we typically see a similar shaped curve, with a clear preference for #1 and a pretty steep drop-off.
It also appears that one of these studies is based on US data and the other on UK data. The "War on Free Clicks" was a WordStream study, if I recall, so their PPC data is generally pretty solid. It also reflects things like paid shopping results and uses a query set that's a bit more commercial. So, it's probably solid, but it reflects a specific niche.
With the diversity of new SERPs, I think it's vitally important to know your own industry landscape. Organic SERPs now have answer boxes, Knowledge Graph, carousels, expanded site-links, in-depth articles, and many features that can radically alter CTR. There's no "average" answer - it's critical to know the space you operate in.
Ultimately, there are two ways to get a sense of CTR data for yourself. Google Webmaster Tools has some clues - I don't completely trust their data, but it's a good starting point. The other option is simply to run some small-scale PPC campaigns and measure yourself. I've done some PPC work in the past in competitive niches, and I'll be blunt - it's all theory until you get out there and try it. CTR also differs radically across ads, based on copy, brand strength, etc.
Each of these problems may have a unique solution, so it gets complicated. Regarding the "design your own" pages, I'm seeing over 5K of those URLs in the search index, and they do probably look very similar. Since these are not the core product pages, I'd strongly consider using META NOINDEX on them. I find that Robots.txt does not do a good job of blocking content that has already been indexed, in most cases. You can add the meta tag dynamically in your code, hopefully, so that just a few lines of code will serve all of these pages.
While these pages aren't "true" duplicates, they look similar enough that, at the scale of your site, they really are diluting your ability to rank. In extreme cases, if you're also serving up product variations, paginated search results, etc., you could even run into Panda issues. Whether or not this is your core problem, from an SEO perspective, cleaning it up can't hurt, and may make it easier to find other problems.
If you can direct message me which campaign you're talking about, I can take a look.
Assuming that your analytics tags are working properly (and that's definitely worth checking), I think you really need to dig in and see what queries you're firing on for these engines and how you're appearing on them. It may be that your page is just a mismatch to the query, or it may be that the title or snippet are misleading in some way.
I don't know of any overarching, general reason this would happen. It's likely to be pretty situational, and you may need to really segment this traffic and see what search users are seeing to find the answer.
Category pages can cause duplicate content, but major categories do often have search value, so it's a trade-off. Typically, it makes more sense to go after duplicate URLs (like product options), search filters, sub-categories, and things like that. It depends a lot on the scope of the problem, though.
My gut reaction is that technical SEO isn't the core problem, though. Ultimately, search traffic doesn't just happen these days. You do need links and social mentions, and you need to actively market and promote yourself to start ranking for non-brand terms. There's no on-page trick to that. Something like schema can help your listings stand out, but it's not going to magically help you rank for terms you don't currently rank on.
Without understanding the site or industry, it's really tough to give advice on where to start, but trying to control how link equity flows through your site only makes sense when you've got a solid amount of link equity to work with. Actually, I wrote about this general issue a while back - you may find it useful:
http://moz.com/blog/whats-better-on-page-seo-or-link-building
JavaScript is client-side code that's generally rendered in browsers, which means Google crawlers typically won't see it, and the almost definitely won't process it for these kinds of directives.
You can create these tags dynamically, but you need to do it with a server-side scripting language, like PHP, ASP, etc. That's a common practice, and many large sites dynamically code canonical tags, META ROBOTS, etc. (I've done it on many sites).
If they've been removed and disavowed, unfortunately there's not a whole lot you can do but wait on the negative SEO front. Moving forward with positive link-building is important - it sounds like part of the problem is that your link profile may have been in a fairly weak position to begin with.
That said, if the removal was long enough ago, you need to make sure that nothing else is going on, like technical issues (including crawl issues), content quality issues, etc. It's not uncommon for link-related problems to be coupled with other issues (you'd be surprised how many sites have been hit by both Panda and Penguin, for example). You don't want to end up in a position where you wait 6 months for the link-related issues to go away, and then realize that wasn't the problem at all (or was only part of the problem).
If you've been hit on all three major engines, my gut feeling is that this goes deeper than a one time attack of bad links.
I'm afraid the answer is "it's complicated" - here's the longer version, from Dr. Matt Peters, our head of Data Science:
"PA and DA are the output from a machine learning model that we then rescale to values between 1-100. The raw output from the model is dimensionless and doesn't have any interesting meaning. The rescaling is linear, but the inputs to the model are rescaled logarithmically before being used in the model. We use the natural log (base e) but the base is pretty arbitrary since one can transform from one base to another by changing coefficients, and the coefficients themselves are set in a regression. The key point is that since the inputs have a log applied to them it is much harder to increase DA from say 70 to 80 then it is from 30 to 40."
Unfortunately, there's no great way to have it both ways. If you want these pages to get indexed for the links, then they're potential duplicates. If Google filters them out, the links probably won't count. Worst case, it could cause Panda-scale problems. Honestly, I suspect the link value is minimal and outweighed by the risk, but it depends quite a bit on the scope of what you're doing and the general link profile of the site.
Sorry, I realized my comments about the 503 were kind of confusing. A 503 shouldn't serve a page for visitors, either - it's just a matter of 404s sometimes seeming a little more friendly, from the user perspective. It just depends on how you're set up.
My only other concern about the 503 is that it's generally intended for short-term use (at least it's been implemented that way). It's great if your site is down for a day and basically tells Google to come back later. If you leave one up for weeks or months, though, I'm honestly not sure what will happen. It's probably going to be treated like a 404, but it also could signal to Google that you have technical problems on the site. So, it may depend on the timeframe. The problem here is that you don't control the malware - it could be weeks before the 3rd party takes action.
So, first off, let me say that this isn't always a bad thing. Less indexed content, if much of your content is duplicated (or nearly duplicated) could actually improve the ranking of the rest of your content. More pages isn't better.
On the other hand, it's one thing when you proactively remove content from the index - it's another when Google does it for you. Are you seeing any patterns? As Chris said, it could be that your internal linking changed, or it could be that Google sees these pages as duplicate or low-quality and has removed them or implicitly canonicalized them (we're seeing more of that - it's like you have a rel=canonical on the page, but you don't). The other issue could be if there are duplicates on other site across the web, causing your copy to be filtered out.
Either way, you have to find out which pages are impacted and figure out the pattern. Do you have XML sitemaps? A well structured set of sitemaps can help you get a better sense of which pages are being indexed.
Potentially, but I'm honestly not sure how Scrid's pages are indexed. Don't you need to log in or something to actually see the content on Scribd?
The other commenters sum it up pretty well for me. It can work, in the short term, but it's also a lot more risky than it used to be. The tactic has just gotten abused way too much. The problem is that, if it fails, it tends to fail hard, and your whole investment just crashed and burned.
If you're looking to get a quick jump in ranking for the very short-term, then I'll be straight with you - it can still work, in some cases. If you're looking to boost an existing business or build something long-term, then I'd stay away from this tactic. You could burn your entire site.
With the canonical/alternate approach, where you're still serving two URLs, the META data is still useful for the mobile crawler (and potentially for mobile search). I'm not sure it's technically "necessary", but I think it's safer all-around.
If the mobile site's URL structure is the same as the desktop site, you probably don't need a separate sitemap, but that sitemap could help you diagnose problems and aid in Google's ability to discover pages for mobile users. I think it boils down to how important the mobile audience is for you and how much you want to invest in it. Not having a separate sitemap isn't going to get you into trouble, it just may be slightly less optimal.
So, it's not indexed, but you're still get malware warnings? That's odd. Honestly, Google shouldn't even see something in a frame as directly being part of your site, in many cases. If you 404 those pages, you potentially kill them for visitors, too - on the other hand, you probably shouldn't be sending your visitors to frames with malware. If it's just a few, I'd probably 404 them temporarily - it'll protect your rankings and your visitors. I'm honestly not sure if the 503 is going to do much that the NOINDEX isn't doing.
If you'd like to Private Message me the campaign, I can take a look. I noticed you have more than one, and I can't find one with 62 notices. The "Notice" level is our lowest-level warning, so sometimes it's just a heads up and isn't cause for concern. The "Tag Value" thing sounds odd, though, so I don't want to tell you that it's fine without taking a closer look.
I endorsed Ade's answer/comments, but I just want to point out something important that people frequently overlook. While rel=canonical and 301 redirects can have a similar impact on SEO, they're completely different for visitors. If a person comes to Site B and there's a canonical tag, they see Site B. Google credits that page to Site A and Site A shows up in search results. If a person comes to Site B and there's a 301-redirect, they go straight to Site A and never see Site B (done properly). These are two completely different experiences with completely different goals. So, set aside the SEO aspect for a minute and ask what you want to have happen to your customers.
The truth is that we're not entirely sure "AuthorRank" even exists, at least not yet, and not in the way most people mean it. Google+ certainly seems to have an impact on personalized rankings, especially time-sensitive (QDF) queries, and authorship mark-up is a nice thing to have, but whether rankings are boosted by the strength of an author's profile is an open question with circumstantial evidence at best, IMO.
Right now, company profiles and author profiles have different advantages. A strong company profile might get a Google+ "brand box" in the SERPs, whereas a strong author profile (or, at least, a properly set up one) could get authorship thumbnails in individual searches. In most cases, though, that's still going to take other signals, like links, and isn't just a result of having a lot of followers.
So, I think it depends on your goal. If you want your business to potentially register "brand" signals ("entity" signals may be a better way to say it), then building your G+ business profile is one piece of that puzzle. If you want to be seen as an influential individual and help your posts rank, then pushing your individual authorship signals and G+ account may be more advantageous. It really depends on what you want to accomplish long-term.
Is there any compelling reason to index this content? It's probably going to look thin to Google, at best (since it's mostly a wrapper around an outside site), and the search value is pretty minimal. In other words, it's good for your users and possibly conversion, but it doesn't have much value for search visitors. If the page is really just a wrapper around a demo site, then I'd consider using META NOINDEX on the frameset and just keep those out of your search results completely.
Sorry you're stuck in that spot. I really would be worried that this "fix" would make life worse for everyone, but it's tough to come up with solutions that don't seem like band-aids. Best you may be able to do is get more aggressive about the de-indexation, focus on improving some core content, and maybe re-work the internal linking to focus more on key pages (and spread internal PR a bit less thinly).
Sorry, didn't notice the date of the original post (apparently got assigned clean-up on this one). Thanks for the update. It's really frustrating trying to sort out what works and doesn't work, especially for something that seems like it should be pretty straightforward. If we can't sort it out, what's your everyday webmaster supposed to do?
Truthfully, we're having a terrible time with authorship here on Moz. Google is showing avatars of commenters and all sorts of weird stuff, and this is without even putting rel-publisher into the mix.
Wish I had some answers, but from what I'm seeing with new Google features, this landscape is evolving so quickly that they haven't really figured out how best to promote and display publisher info. Trying to tweak the signals to remove authorship could backfire, so I'd honestly probably let it ride for a couple of months unless it's a major issue for the brand. I don't know of any way to tell Google to remove a rich snippet.
Technically, it should be ok, but this is a scenario that behave unpredictably, and it's why many mobile SEO folks seems to be recommending more and more moving away from alternate domains. You're doing it the way Google recommends for your situation, as far as I'm aware, but I would monitor the situation very, very carefully, and make sure that there's not crossover or duplicate content issues.
It's a bit technically challenging, but you may also want to make sure that Googlebot-Mobile is hitting the proper site. See this advice:
https://developers.google.com/webmasters/smartphone-sites/redirects
Given their history, two domains with overlapping content and a similar name seems like a terrible idea to me, to be blunt. If this really is a Panda issue, then you're potentially going to aggravate the situation and send out even more low quality signals.
It's hard to speculate, but I've seen a few situations where what seemed like Panda turned out to be something deeper. Directory clients have been hit hard, for example, as Google just seems to be devaluing the entire space (along with price comparison sites, many types of affiliates, etc.). I'm not talking about spammy sites, even, but the ones that provide some original value. It's just that Google doesn't see them as the end-supplier, and so they're getting discounted.
An end-run to a new domain isn't going to fix this. I strongly suspect that you've got something deeper going on that may take a radical overhaul of the main site and even the business/brand. I think it's better to accept that now than continue a gradual decline over the next couple of years.
I tend to agree with Sorina - in a perfect world, it would be great if you could somehow vet that content and make sure it stays safe for your users (even crawl the sites offline if you need to). What content does the page have around the iFrame (if you can explain it generally without giving away private details)? I'm wondering if these pages should be indexed at all, malware or no, since they're mostly just re-displays of other people's content. META NOINDEX might be a better bet here.
Ah... you're saying have a "View All" page but then not canonical to it? I guess my only concern about that is that then you've got another crawl path and possible duplicates. In that case, you might want to Noindex the "View All" and only have it available to users. It depends a lot on the scope of pages we're talking, as always.
Yeah, it gets ugly fast, and even done "by the book" you're often going to need to monitor your index and make adjustments, I've found. That said, the official Google stance (at least the last I heard) is that you should canonical to the page with no parameters and rel=prev/next to the parameterized versions (your 2nd-to-last example):
See the bottom of this Google blog post:
http://googlewebmastercentral.blogspot.com/2011/09/pagination-with-relnext-and-relprev.html
The other option would be to use rel=prev/next on the paginated URLs and then dynamically Meta Noindex anything with parameters. Honestly, it really depends on what works, and it can take a while to sort out. Also, keep in mind that Bing doesn't handle rel=prev/next quite the same way as Google.
Just one note here - I generally wouldn't use "View All" and rel=prev/next. It's a bit of a mixed signal. If you can create a friendly, fast-loading "View All" page, then rel=canonical the paginated URLs back to the "View All" page.
Agreed, though, that your Nofollow, Noindex is basically overriding the rel=prev/next. I've honestly heard mixed signals from people (including prominent SEOs who handle very large media sites) about how effective rel=prev/next is. I think Meta-robots is a stronger signal, so if you're really worried about duplicates, it's probably doing fine. If you want page 3 of 8 (for example) to rank for some reason, then rel=prev/next opens up that possibility, but it may also be a bit weaker cue in terms of duplication. It's a bit of a trade-off. If your currently approach is keeping pages out of the index, I'd probably leave it alone.
I know many SEOs who have gone through the process and preach the necessity of an honest and thorough reconsideration request. The more details you give, the more they're likely to take it seriously.
I haven't heard that it's necessary to throw anyone under the bus, but it probably is true that if you're only apologizing because you got caught, they might not be so sympathetic. If someone actually did damage without your knowledge (like an old agency), then I'd come clean about that, but don't go off ranting about how it's all their fault. If you hired them, be honest about it. I definitely wouldn't make up a story for sympathy, as that can go wrong fast. These requests, if they get through the first filter (I suspect), are reviewed by humans, and they have a much better ability to detect BS than the algorithm does.
This process can take a painfully long time, even done right, but I do have a couple of concerns:
(1) Assuming I understand the situation, I think using Robots.txt on top of 301-redirects is a bad idea. If Google doesn't recrawl the pages, they won't process the 301s, and Robots.txt is bad for removal (good for prevention, but not once something is in the index). Basically, you're telling Google not to re-crawl these pages, and if they don't re-crawl, they won't process the 301s. So, I'd drop the Robots.txt blocking for now, honestly.
(2) What's your internationalization strategy? You could potential try rel="alternate"/hreflang to specify US vs. UK English, target each domain in webmaster tools, and leave the duplicates alone. If you 301-redirect, you're not giving the UK site a chance to rank properly on Google.co.uk (if that's your objective).
I'd proceed very carefully. Here's the problem - those same links from copies of your content that could potentially be causing you trouble with Penguin are also helping Google understand that you're the original source of that content. You could remove those links, still have Penguin in play and start getting outranked by the copies or even get hit by Panda on top of Penguin.
I would really want to see your entire link profile, because these kinds of links by themselves aren't that likely to cause a Penguin penalty. Obviously, your webmaster knows the site and I don't, buy my gut says that something else is going on here and that you've got other, more problematic links in play.
If these links are the problem, it may be better to disavow them in Google Webmaster Tools. That could help maintain the signal that these are copies but remove any negative aspects of the links themselves. Unfortunately, it's really situational and depends on the scope of the problem and your overall link profile.
I'd just warn that Google seems to be signalling that toolbar PR updates may slow down or eventually stop altogether. If it wasn't for industry outcry, they would've stopped a while ago, I'm sure.
I don't know anyone who has a back-channel into when Google updates toolbar PR. They've been increasingly inconsistent. The last one was in February, as best I'm aware. Shaun Anderson reports them pretty reliable, from what I've seen:
http://www.hobo-web.co.uk/google-pr-update/
That said, I agree with William - toolbar PR is a very unreliable metric and limited metric, and it's often so outdated by the time it updates that it's easy to misinterpret. Real PageRank is being updated in nearly real-time, and even "true" PR is just one of 200+ ranking signals.
Days, weeks, months, or sometimes never It is a very inexact science, and we're unaware of the vast majority of the 500+ updates Google makes every year.