Welcome to the Q&A Forum

Hi Jim,

From my own experience with Panda-impacted sites, I've seen good results from applying meta robots "noindex" to URLs with thin content.  The trick is finding the right pages to noindex.  Be diligent in your analytics up front!

We had a large site (~800K URLs), with a large amount of content we suspected would look "thin" to Panda (~30%).  We applied the noindex to pages that didn't meet our threshold value for content, and watched the traffic slowly drop as Google re-crawled the pages and honored the noindex.

It turned out that our analytics on the front end hadn't recognized just how much long-tail traffic the noindexed URLs were getting.  We lost too much traffic.  After about 3 weeks, we essentially reset the noindex threshold to get some of those pages back earning some traffic, which had a meaningful impact on our monetization.

So my recommendation is to do rigorous web analytics up front, decide how much traffic you can afford to lose (you will lose some) and begin the process of setting your thresholds for noindex.  It takes a few tries.

Especially if you value the earning potential of your site over the long term, I would be much more inclined to noindex deeply up front.  As long as your business can survive on the traffic generated by those 1000 pages, noindex the rest, and begin a long-term plan for improving content on the other 8000 pages.

Have you checked your GA implementation?

1. Go into Admin > Property > Property Settings

• Is your Default URL set correctly?  http vs https?  www vs non-www?

2. Go into Admin > View > View Settings

• Is your Default URL set correctly?  http vs https?  www vs non-www?

3. Have you crawled the site with Screaming Frog to verify your tag is on every page?

• http://www.seerinteractive.com/blog/screaming-frog-to-verify-google-analytics/

This "tip" may seem shockingly obvious, but I've been blown away by how many times I've seen it go wrong.

Hi Robert,

You're doing the right thing!

1. Ask the right questions (you are)
2. Ask them in a reputable community (like this)
3. Take the combined weight of your own experience and good feedback from the Moz community to your client (your next step)

You should expect to get a blizzard of counter-argument and obfuscation from the link development company.  These days, it's very likely that the rep in question spends a lot of time on the phone trying to explain away the fact that his company has a "Kick Me" sign on its back, and that his comany's activities put their clients at risk.  He's just trying to stop the bleeding.

Your advice to your company is directed toward making sure that your website is in the best possible position to earn your company money over the long term.  The rep is just trying to keep yet another client from cancelling on him - which is a goal not aligned with the long-term health of your website.

Hi Fabrizio,

Since you already have an OSE account, just enter that URL, and then click on the "Just-Discovered beta" tab on the far-right.  You'll see 350 links to that URL.

Alternately, ahrefs has 195 domains that link to the URL you provided, but of course you'll need an account to see all the data.

Hope that helps!

Matt Cutts has stated that within Google, "Web Spam and Search Quality does not use Google Analytics in ranking.  It won't affect your ranking within Google search results".  You can watch the Google Webmaster Help video here: http://www.youtube.com/watch?v=LLmO1GE4GvI

He made a point of qualifying his answer at the end by saying that his answer was accurate "as of today, May 17, 2010", so take that how you will

Hi Jordan,

It's definitely worth your while to get those 301s in place.  2K URLs is quite a small batch of redirects, in the larger scheme of things.  I've worked with sites that have had to redirect millions of URLs, and because they were planned, QA'd, and executed correctly, they went off without a hitch.

By correctly executing these redirects, you're actually helping engines maintain a more orderly index, so they will not "ignore them", to your question.

I think the trickiest aspect of the redirect plan is that the subdomain URLs are going to go 404 after the redirects are in place.  Ideally, 301 redirects should remain in place in perpetuity, but having them in place for 3 months should be more than adequate to get them crawled, and have engines recognize the redirects.

Since you've said that there aren't that many external links to these subdomain URLs, you won't be doing this redirection for the link authority boost, rather to maintain a good user experience for people who have previously visited the URLs, and attempt to go back to them.  If 80% of your traffic is PPC, you'll be surprised how many people have visited via paid search, bookmarked the URL, and then go back to it.  Technically that's a Direct visit, but analytics will likely track it as a paid search clicks, since the URL will have all of the paid search parameters appended.

There's no down-side to getting the 301 redirect from subdomain to sub-folder in place, and keeping them in place until the subdomain URLs go 404.  Just make sure that you're doing a good job of creating a 1-to-1 mapping of the redirects.

Hope that helps!

Hi Ben,

The answer to your question boils down to usability and link equity:

• Usability: Did the old URLs get lots of Direct and Referring traffic?  E.g., do people have them bookmarked, type them directly into the address bar, or follow links from other sites?  If so, there's an argument to be made for 301 redirecting the old URLs to their equivalent, new URLs.  That makes for a much more seamless user experience, and increases the odds that visitors from these traffic sources will become customers, continue to be customers, etc.
• Link equity: When you look at a Top Pages report (in Google Webmaster Tools, Open Site Explorer, or ahrefs), how many of those most-linked and / or best-ranking pages are old product URLs?  If product URLs are showing up in these reports, they definitely require a 301 redirect to an equivalent, new URL so that link equity isn't lost.

However, if (as is common with a large number of ecommerce sites), your old product URLs got virtually zero Direct or Referring traffic, and had virtually zero deep links, then letting the URLs go 404 is just fine.  I think I remember a link churn report in the early days of LinkScape when they reported that something on the order of 80% of the URLs they had discovered would be 404 within a year.  URL churn is a part of the web.

If you decide not to 301 those old URLs, then you simply want to serve a really consistent signal to engines that they're gone, and not coming back.  Recently, JohnMu from Google suggested recently that there's a tiny difference in how Google treats 404 versus 410 response codes - 404s are often re-crawled (which leads to those 404 error reports in GWT), whereas 410 is treated as a more "permanent" indicator that the URL is gone for good, so 410s are removed from the index a tiny bit faster.  Read more: http://www.seroundtable.com/google-content-removal-16851.html

Hope that helps!

Hi Sara,

The best way to check what Googlebot is seeing is by going into your Google Webmaster Tools account, and clicking

Health >>Fetch as Googlebot.

Paste your URL in, and then click on whatever appears in the "Fetch Status" column, which will likely be "Success".  You can then see the code that Googlebot is reading, which should be your page source code.

If you don't have GWT set up yet, you can accomplish pretty much the same thing with publicly available spider simulators like this one: http://www.webconfs.com/search-engine-spider-simulator.php

Hope that helps!

Hi wooz,

You're absolutely correct - implement 301 redirects from your old URLs to their equivalent URLs on your new domain, and then declare your "Change of Address" in Google Webmaster Tools.

There's quite a bit more detail covered here: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=83105

Matt Cutts has said that there's no loss or leakage of link juice through 301 redirects: http://searchengineland.com/google-pagerank-dilution-through-a-301-redirect-is-a-myth-149656

The main issues that I've seen with sites losing traffic due to domain migrations are usually due to one or both of the following factors:

1. 301 redirects are not properly set up.  Only a small portion of URLs from the old domain are correctly 301 redirected to their equivalents on the main domain.  As a result, old URLs that have incoming links go 404, and the link equity is not transferred to the new domain.
2. The new site has vastly different information architecture than the old site.  It is larger, structurally different, or targets new keyword spaces.  As a result, there is not a 1:1 correlation between the old site and the new site, and the new domain can't compete as effectively in the new keyword spaces as the old domain could compete in the previous keyword spaces.

Google is generally quite quick at picking up the Change of Address notification and applying it to your new domain.  There shouldn't be any "lag time" where your old domain has disappeared from the SERPs, and has not been replaced by your new domain.

Hope that helps!

Having that many links in your global nav can hurt you from an SEO perspective.  You're diluting the link authority that any given page is able to pass to another page by splitting it across 175 links - and that's before you get to links in the body copy, right nav, left nav, and footer.  Essentially, you'll have a site with a large number of pages that each have a tiny amount of link authority.  It's great for indexation, but bad for ranking.

Unless you're operating in keyword markets where a tiny amount of link authority is all that's necessary to compete for the super long-tail phrases you're targeting, that means that most of your pages will be unable to compete for shorter-tail terms.

You have to prioritize those 175 links and think about the business value of each of the pages that you're linking to, and the competitiveness of the keyword market that each page addresses.

• If you have lots of lucrative head terms in that 175, but you know you won't be able to rank for the foreseeable future because they're so competitive, eliminate them from the global nav.
• Similarly, if you have lots of obscure long-tail terms in the 175, you may be able to rank very well immediately, but the page will only get one visit per month, even with a #1 ranking.  Eliminate those as well.  .

Balance the conversion value of each page with its ability to rank for the term.  The terms that strike that balance deserve to be in your global nav.

And think about this from your users' perspectives as well - 175 links in a global nav is a usability nightmare, and will cost you sales down the road, because your users will be overwhelmed.  There's a saying in conversion rate optimization that "by emphasizing everything, you emphasize nothing."  175 links fits that description.

Hope that helps!

You're correct that it's a GA hack.  Avoid it.

Google has publicly stated that they don't use your site-specific GA metrics to influence organic search rankings.  E.g., they're not taking data from your GA profile, and feeding that to the Search Quality team to determine if your site should rank better or worse.  They have MANY better ways to accurately track anonymous user interactions with sites at scale (e.g. Chrome).

The only thing that you'll accomplish with this code is making all of your own internal metrics turn to garbage.  Accurate metrics are important.  If you bounce rate is high, knowing that allows you to take action to improve your site and reduce it.

The more people who stay on your site for more than 1 pageview, the more money your business is likely to make. Improve your bounce rate to improve the profitability of your website, not for some supposed correlation between bounce rate and organic search ranking.

Hi LilyRay,

Regarding your Penguin penalization, I would treat it like any other pre-Penguin link-based penalty.  I have worked with many sites that have been penalized for manipulative linking, and the process to get the penalty lifted is always the same:

1. REMOVE as many of the manipulative links as you can.  It's the link that Google has classified as manipulative.  The anchor text was just the identifier that helped them find it.  Changing the anchor text of a manipulative links and leaving them up will keep the penalties associated with those links in place.

Document all of the steps that you're taking to eliminate manipulative links.  Make a neat, bulleted list, with the link(s), network(s), actions taken by you, and the results.  In some cases, you won't be able to remove a link.  That's understandable, as they're not in your control.  While you're at it, clean up ANYTHING else on your site that could be perceived as on-page spam.  You're trying to prove to Google that you are a good citizen of the web, so make your site as sparkly as you can.

Once you've completed these steps, submit all of your documented work as part of your reconsideration request, to show Google that you're operating in good faith.  Under normal circumstances, wait times for reconsideration requests can be anywhere from a week to a month.  With the mass of reconsiderations that Google is getting right now, I'd expect a longer wait.

I'm sure this process sounds painful, and it is, but it's the only way to get back from a penalty that I've seen be effective.

Hi Christian,

It's good that you got confirmation that no manual action has been taken against the site.  That means that changes in traffic are most likely due to changes in how Google is calculating individual relevance or authority signals:

1. Relevance - Since a large percentage of your traffic was coming from Image Search, I took a quick look at your source code.  You are using the image title attribute, but you are not using the <alt>attribute on your images.  The <alt>attribute is a much stronger relevance signal to engines.  Make sure to include descriptive alt text on all of your photos, for example:</alt></alt>

class="size-thumbnail wp-image-2824" title="Nimitz young lieutenant" src="http://general-history.com/wp-content/uploads/2012/09/Nimitz-young-lieutenant-141x150.jpg" alt="Chester William Nimitz, photographed as a Lieutenant, United States Navy" width="141" height="150" />

2. Authority - Because your site is built on Wordpress, there's a chance that the photos (or posts) that were getting the majority of your traffic are being pushed farther and farther away from the home page as new posts are added. As they're pushed successive clicks away from the home page, the link authority flowing to them drops drastically.  A quick test you can run is to grab the 3 posts that were getting the most traffic, and put them in a "Most Popular Posts" navigation widget on the home page.  That should increase the flow of link authority to them substantially, and should allow them to start ranking again (if this is the problem).

Another possibility is that your site's links got devalued by Penguin.  But I'd address #1 and #2 before jumping to that conclusion.  Hope this helps!

Hi wooz,

You're absolutely correct - implement 301 redirects from your old URLs to their equivalent URLs on your new domain, and then declare your "Change of Address" in Google Webmaster Tools.

There's quite a bit more detail covered here: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=83105

Matt Cutts has said that there's no loss or leakage of link juice through 301 redirects: http://searchengineland.com/google-pagerank-dilution-through-a-301-redirect-is-a-myth-149656

The main issues that I've seen with sites losing traffic due to domain migrations are usually due to one or both of the following factors:

1. 301 redirects are not properly set up.  Only a small portion of URLs from the old domain are correctly 301 redirected to their equivalents on the main domain.  As a result, old URLs that have incoming links go 404, and the link equity is not transferred to the new domain.
2. The new site has vastly different information architecture than the old site.  It is larger, structurally different, or targets new keyword spaces.  As a result, there is not a 1:1 correlation between the old site and the new site, and the new domain can't compete as effectively in the new keyword spaces as the old domain could compete in the previous keyword spaces.

Google is generally quite quick at picking up the Change of Address notification and applying it to your new domain.  There shouldn't be any "lag time" where your old domain has disappeared from the SERPs, and has not been replaced by your new domain.

Hope that helps!

Hi Anthony,

I use a combination of tools for audits.  SEOmoz is great for client-facing reports, and tracking issues over time.  The downside is that you don't have that "on-demand" capability to crank out a full audit the instant you need it.

For on-demand audits, I use Screaming Frog, which is free for 500 URLs, and $99 for an unlimited license - it's worth every penny, and returns a full range of technical SEO data, which you can export and manipulate in Excel.

If you're looking for something with a lot more SEO-specific functionality, it's definitely worth trying IIS Search Engine Optimization Toolkit 1.0:

http://www.iis.net/download/seotoolkit

Have you checked your GA implementation?

1. Go into Admin > Property > Property Settings

• Is your Default URL set correctly?  http vs https?  www vs non-www?

2. Go into Admin > View > View Settings

• Is your Default URL set correctly?  http vs https?  www vs non-www?

3. Have you crawled the site with Screaming Frog to verify your tag is on every page?

• http://www.seerinteractive.com/blog/screaming-frog-to-verify-google-analytics/

This "tip" may seem shockingly obvious, but I've been blown away by how many times I've seen it go wrong.

Hi Nathan,

The delta between the number of pages returned by the site: operator and the number of pages in your sitemap could be down to a number of issues:

1. Your XML sitemap may represent only a percentage of the total number of valid content URLs that your site is capable of generating.

a) Often sites will only generate XML sitemaps for URLs that someone has decided are "important", when the total number of URLs is much larger.

2. Your XML sitemap contains ALL the valid content URLs that your site is capable of generating, but search engines are somehow finding more URLs.

a) Look in Google Webmaster Tools under Optimization >> HTML improvements >> Duplicate title tags

i) Do the pages with duplicate titles have duplicate page content?  If so, your publishing platform is allowing multiple URLs to render the same content, which is a bug that needs to be fixed

b) Run a crawler like Xenu Link Sleuth or Screaming Frog against your site, and see how many URLs they discover.  Export the results to Excel and look for weird URLs

i) Usually culprits for duplicate content include incorrect canonicalization (www vs non-www, URLs ending in /index.html vs just /, etc)

ii) Look for URLs ending with strange query strings (affiliate tracking, session IDs, etc)

c) Use the site: operator in other engines (Bing, blekko, etc) and compare the numbers they return.  Especially if this number is larger than the number Google is returning, starting looking for weird URL patterns

Also, I'm not sure what you mean by "the domain canonical has been set correctly".  If you're referring to use of the canonical link element for every URL, there are plenty of ways that can go wrong.  E.g., if your CMS requires that each published URL have rel="canonical", but allows URLs to be published with and without the trailing /index.html, you can end up with a canonical link element on the non-canonical version of the URL, further confusing engines.  Something to look into.

Hi Allie,

You're absolutely on the right track.  Cross-domain canonicalization was created to deal with situations exactly like the one you're describing.  I've used it many times, and it works great.

http://googlewebmastercentral.blogspot.com/2009/12/handling-legitimate-cross-domain.html

Hi Ben,

The answer to your question boils down to usability and link equity:

• Usability: Did the old URLs get lots of Direct and Referring traffic?  E.g., do people have them bookmarked, type them directly into the address bar, or follow links from other sites?  If so, there's an argument to be made for 301 redirecting the old URLs to their equivalent, new URLs.  That makes for a much more seamless user experience, and increases the odds that visitors from these traffic sources will become customers, continue to be customers, etc.
• Link equity: When you look at a Top Pages report (in Google Webmaster Tools, Open Site Explorer, or ahrefs), how many of those most-linked and / or best-ranking pages are old product URLs?  If product URLs are showing up in these reports, they definitely require a 301 redirect to an equivalent, new URL so that link equity isn't lost.

However, if (as is common with a large number of ecommerce sites), your old product URLs got virtually zero Direct or Referring traffic, and had virtually zero deep links, then letting the URLs go 404 is just fine.  I think I remember a link churn report in the early days of LinkScape when they reported that something on the order of 80% of the URLs they had discovered would be 404 within a year.  URL churn is a part of the web.

If you decide not to 301 those old URLs, then you simply want to serve a really consistent signal to engines that they're gone, and not coming back.  Recently, JohnMu from Google suggested recently that there's a tiny difference in how Google treats 404 versus 410 response codes - 404s are often re-crawled (which leads to those 404 error reports in GWT), whereas 410 is treated as a more "permanent" indicator that the URL is gone for good, so 410s are removed from the index a tiny bit faster.  Read more: http://www.seroundtable.com/google-content-removal-16851.html

Hope that helps!