Welcome to the Q&A Forum

I would not nofollow "everything" on your site, as in every outbound link, affiliate or not. However, if you can find a plugin that will nofollow every "affiliate" link, you could go back and manually remove the nofollow link on those you wish to editorially endorse - though "technically" you shouldn't have a financial arrangement with them.

Much of the time affiliate links go through some sort of redirect that blocks pagerank anyway, such as a 302, or a redirect to a blocked folder, etc... This is typical with most of the larger affiliate networks.

Hello Rubix,

Saijo gave you some great advice, but I'm concerned about the fact that you have that page in the first place, and that it produces those URL parameters. It suggests to me that instead of showing a 404 error on the contact-office.aspx page (assuming that pages doesn't exist on that URL) you are redirecting the user who tries to access that URL to the /404.html page (e.g. /404.html?aspxerrorpath=/contact-office.aspx).

Typically you want the 404 http status code to show on the URL the user is trying to unsuccessfully access. In this case instead of redirecting them to your "404 page URL" you would want to show your customized 404 message (and ensure it returns a 404 status code, use this tool) on www.yourdomain.com/contact-office.aspx.

I hope this makes sense to you. If not, feel free to ask for clarification.

Sometimes you need to leave the crawl path open to Googlebot so they can get around the site. A specific example that may be relevant to you is in pagination. If you have 100 products and are only showing 10 on the first page Google will not be able to reach the other 90 product pages as easily if you block paginated pages in the robots.txt. Better options in such a case might be a robots noindex,follow meta tag, rel next/prev tags, or a "view all" canonical page.

If these pages aren't important to the crawlability of the site, such as internal search results, you could block them in the robots.txt file with little or no issues, and it would help to get them out of the index. If they aren't useful for spiders or users, or anything else, then yes you can and should probably let them 404, rather than blocking.

Yes, I do like to leave the blocked or removed URLs in the sitemap for just a little while to ensure Googlebog revisits them and sees the noindex tag, 404 error code, 301 redirect, or whatever it is they need to see in order to update their index. They'll get there on their own eventually, but I find it faster to send them to the pages myself. Once Googlebot visits these URls and updates their index you should remove them from your sitemaps.

Hello Segafredo,

I have seen it before, but not very often because, as Chris said, it just isn't a good format for URLs. As to why it is happening, it would help if we know what kind of site you have. Is it on a CMS? Is it PHP, CFM, ASP...

Hello AshShep1,

I don't think anyone is going to be able to tell you for sure whether Google looks at LSI specifically on a per-page or site-wide basis. However, it sounds like there is definitely some overlap with your landing pages. For example, "funeral planning" and "funeral plans" could probably be one page, which then links off to the others. The fact that Google bolds "planning" when I search for "funeral plans" tells me they view the two keywords as being so semantically related as to be almost synonymous.

The "funeral plan costs" and "paying for a funeral" page sound like they could be combined, and the "compare funeral plans" and "prepaid funeral plans" could probably be combined. I would look into the keywords to see which, if any, have the overwhelming majority of search volume and focus on that one while using the others as secondary or tertiary keywords within the context of the page.

But...Really it comes down to how you think the information is best presented to your visitors. If the "prepaid funeral plans" or the "compare funeral plans" content is best presented on its own then that is what you should do.

You may be seeking a more concrete answer from someone so I will leave this question open as a discussion for awhile. Please let me know if I can be of any assistance.

Adrian,

I agree that there certainly is a right answer to the question posted, as the question asks specifically about one way to manage the issue, being a block of filters in the robots.txt file. What I was getting at is that this may or may not necessarily be the "best" way, and that I'd need to look at your site and your unique situation to figure our which would be the best solution for your needs.

It is very likely that with these parameters a robots.txt file block is the best approach, assuming the parameters aren't added by default into category page or category pagination page navigational links, as then it would affect the bot's ability to crawl the site. Also, if people are linking to those URLs (highly unlikely though) you may consider a robots meta noindex,follow tag instead so the pagerank could flow to other pages.

And I'm not entirely sure the code you provided above will work if the blocked parameter is the first one in the string (e.g. domain.com/category/?color=red) as there is the additional wildcard between the ? and the parameter. I would advise testing this in Google Webmaster Tools first.

1. On the Webmaster Tools Home page, click the site you want.
2. Under Crawl, click Blocked URLs.
3. If it's not already selected, click the Test robots.txt tab.
4. Copy the content of your robots.txt file, and paste it into the first box.
5. In the URLs box, list the site to test against.
6. In the User-agents list, select the user-agents you want (e.g. Googlebot)

You are assuming correctly. The spider will see the content that is shown by default, which in your case should be the retail pricing.

These look like simple broken links, often caused by using relative paths incorrectly or forgetting to include http:// when linking to a website (e.g. which will then make the link look like this to a spider http://www.yourdopmain.com/www.someotherdomain.com.</p>
<p>In your case it looks like there are some HTML URL encoding issues too. <strong>%E2 %80 %9D</strong> is a string used to encode a right quotation mark, and <strong>%E2 %80 %9C</strong> is a string used to encode a left quotation mark.</p>
<p>I'm going to assume that your content management system is adding this code, possibly when content is being copied and pasted from another CMS, word processing software, website builder, etc...</p>
<p>You can fix it by replacing that code with a standard ASCII )

  which will then make the link look like this to a spider http://www.yourdopmain.com/www.someotherdomain.com.</p>
<p>In your case it looks like there are some HTML URL encoding issues too. <strong>%E2 %80 %9D</strong> is a string used to encode a right quotation mark, and <strong>%E2 %80 %9C</strong> is a string used to encode a left quotation mark.</p>
<p>I'm going to assume that your content management system is adding this code, possibly when content is being copied and pasted from another CMS, word processing software, website builder, etc...</p>
<p>You can fix it by replacing that code with a standard ASCII )

Alternatively you could do this an entirely different, and better IMHO, way by putting all of that content ON the page and using div tags with "display:none" that will change to display the content when the link is clicked, activating an on-click javascript event. This sounds complicated but really it is what most eCommerce stores do when they have tabbed content (e.g. product description, details, specs, shipping....) that doesn't all display at once and requires a user to click a tab to show the content there.

Here is a simple example with information on how to implement it.

I assume one of the places this issue can be found is on your About page where the visitor can learn more about each member of the team.

I am not a developer and am going to speak about this to the best of my knowledge. It may not be "technically" accurate, but I hope it will be more helpful to you than complicating the situation...

The URL being linked to, such as .../194 should probably be /#!194 instead, which would allow you to change the content via AJAX without changing the URL of the page (as a hashbang URL is treated much like an internal page link, known as a named anchor, which takes you to a different part of the page instead of a new page/URL).

Please see https://developers.google.com/webmasters/ajax-crawling/docs/getting-started for more information on implementing Google-friendly, crawlable content with AJAX.

I will leave this question open for discussion in case others have more to add.

Blocking the /ads/ directory via the robots.txt file is not going to stop your site from being associated with affiliates who link to you. The best thing you can do in this case would be one of two things:

1. Ensure all affiliates are linking with a rel = "nofollow" attibute in the href tag (i.e. nofollow their links). You can provide the nofollow tag in the link code that they copy to make it easy for them.

Or

2. If you are trying to get some pagerank out of your affiliate links, but want to be able to react quickly in the event of a link-based penalty as a result of this practice, you could have affiliates link through another domain, which then 301 redirects to your landing page. This way you can kill off all the links simply be cutting the redirect (or changing it to a 302, or blocking Google from following the redirect with a tactic similar to the robots.txt block described above...) on the other domain instead of having to disavow a bunch of affiliate links, or instead of asking affiliates to updating their links.

Personally I'd go with option #1. You can combine that with the robots.txt block of your /ads/ directory too.

Hello Carl,

Let us know how your fixes work out for you. Also, be sure to read these articles, which will give you a good idea of your options so you can choose the best one/s for your situation:

Building Faceted Navigation That Doesn't Suck Faceted Navigation Whiteboard Friday
Duplicate Content: Block, Redirect or Canonical
Guide to eCommerce Facets, Filters and Categories
Rel Canonical How To and Why Not
Moz.com Guide to Duplicate Content

If you feel you could use more help with this just let us know. Thanks!

Hello Adrian,

The Moz reports are meant to help you uncover issues like this. If you're seeing non-canonical URLs in the Moz report then there is a potential issue for Google, Bing and other search engines as well.

Google does respect wildcards (*) in the robots.txt file, though it can easily be done wrong. There is not right or wrong answer to the issue of using filters or faceted navigation, as each circumstance is going to be different. However, I hope some of these articles will help you identify the best approach for your needs:

(Note: Faceted Navigation is not exactly the same as category filters, but the issues and possible solutions are very similar

)Building Faceted Navigation That Doesn't Suck Faceted Navigation Whiteboard Friday
Duplicate Content: Block, Redirect or Canonical
Guide to eCommerce Facets, Filters and Categories
Rel Canonical How To and Why Not
Moz.com Guide to Duplicate Content

I don't know how your store handles these (e.g. does it add the filter automatically, or only when a user selects a filter?) so I can't give you the answer, but I promise if you read those articles above you will have a very good understanding of all of the options so you can choose which is best for you. That might end up being as simple as blocking the filters in your robots.txt file, or you may opt for rel canonical, noindex meta tag, ajax, Google parameter handling, etc...

Good luck!

Selamat pagi Juergen!

If you are using a content management system (e.g. Drupal, Wordpress...) or a popular eCommerce solution (WooCommerce for Wordpress, 3dcart, Magento etc... there are usually already plugins/add-ons that will do this for you. For example:

Social Checkout plugin for WooCommerce

Some CS Cart Options

3dCarts's Facebook Connect add-on

Share Your Purchase extension for OpenCart

BigCommerce Social Sharing / Co-Buying Feature

Magento Connect AddShoppers Social Commerce Extension

Etc...

If this is a totally custom site you will probably need to hire a developer to set it up because, as Lesley mentioned above, it isn't as simple as copying and pasting something into your page. However, it isn't that difficult either and a decent developer should be able to get this done for you in a day.

I will leave this open as a discussion for further comments and ideas.

This sounds like Verizon's "Search Assist" page, which is shown to Verizon users when they miss-type a URL or try to access a domain that doesn't exist. How this happened to spike is anyone's guess. Here are two likely possibilities...

1. The browser hijack mentioned by Mike Roberts above could be trying to send Verizon users to a domain that doesn't exist, but is similar to yours (e.g. yourdomain.com and yourdomain.info) so when their browser tries to access the domain Verizon redirects them to their "Search Assist" page with similar results, including yours.

2. A company similar to yours, or perhaps even your own, could have been featured on a television show (e.g. Good Morning America) with a large viewing audience that isn't super web-savvy. Instead of typing YourBrandDomain.com into the URL bar they might think of it as a search bar and type "Your Brand" instead, resulting in the redirect to Verizon's "Search Assist" page of search results, which lists your domain as one of the results.

Hello Nicolas,

You're welcome! I am glad you found the answer helpful. I direct the SEO strategy for seOverflow, and yes we do work with small businesses. Fill out our contact form and we'll be in touch. You can also look into other reputable SEO agencies and freelancers listed here on Moz.com.

Nice site.

Hello Pikka,

I searched for that page on Google and see the review rich snippet in the SERPs. Perhaps you are thinking this is going to change your ranking. It is not. It will change the way the search engine result looks, which will encourage a better click-through-rate, which "could" indirectly improve rankings, but technically speaking at this point in time there is no direct ranking benefit from having a review/rating rich snippet in the SERPs.

They look good in the search results: See here. Be happy!

You can try these tips if you haven't already:

Optimize the image filename.

Have another look at the alt text for each image and ensure it is optimized.

Link to the image from the image. For example:

Make sure your images are big and beautiful.

Differentiate your images from others if they are not exclusive to your site. For instance, you could resize them, change the hue a little, run auto-levels on it, mirror it, crop it, etc...

Consider submitting an image sitemap.

Hotlink to the image from a guest post on another site.

Good luck!

I would use a 301.

Reason #1 - Some of those profiles are going to have external links, as Robert stated in his reply.

Reason #2 - A 302 will not remove the URL from the index like a 301 will. Over time you could end up with many more temporary redirects than actual pages, which may look pretty bad. I prefer not to bloat the index with URLs like that.

In the event that someone comes back you can remove the 301 redirect with no ill-effect. If it has been in place for a long time (e.g. a year or more) the profile page may not get its pagerank back from those older links, which would be fine.

Please also keep in mind that Google can choose to treat a 301 like a 404 if the content doesn't match up. This could easily happen with the situation you mention above. However, I don't really think there is any other option other than doing the 302. I'd  rather let it be treated as a 404 than a 302 if we're talking about user-generated pages here because they can get out of hand and bloat the index over time.

eCommerce sites that have products coming and going all the time have similar issues. Imagine if every product a store ever sold stayed indexed for years after the product was no longer available. Even if users are redirected I don't think this is an ideal situation.

Disabling the feature will keep the trackback link from being posted on your site, but it wouldn't remove existing links from other sites. However, it may discourage people who are linking to you just to get the trackback link. This could be good or bad, depending on how you look at it. Personally, I don't' want or need any links coming from someone who is doing it just for the trackback. Most of them are nofollowed anyway.

Typically if Google is choosing to show a snippet of content instead of your meta description then there is something they don't like about your meta description. For instance, it could be too short, too long, over-optimized, not formatted correctly, etc...

You can't force Google to use your meta description, but you can play around with rewriting meta tags to see if they end up liking one enough to use when someone searches for your primary keywords on that page.

Also use the No ODP tag if you aren't already.

I'm going to go against the grain here and say that I don't think you need or want separate pages. These are synonyms. The phrases are obviously semantically related. When I type "elliptical machine" and search on Google I see results with the world "elliptical trainer" in bold as well. If it were me and I had one category page to optimize I would title it something like:

Elliptical Machines - StoreName Has the Best Elliptical Trainer Brands in Stock for Less!

Or some variation that allows you to get both of those two-word phrases in there once.

If you do have the opportunity to optimize more than one page, and both are equally relevant to the searches (e.g. Home page and category page), then I agree that you should target one per page, though I would intersperse the other variation here and there on each page too.

Product pages are likely going to rank best for the brand/model of machine/trainer so you can just write naturally and use each keyword interchangeably (e.g. The BrandX elliptical machine is an excellent value, combining affordability with the features, style and durability of the more costly elliptical trainers on the market...)

Hello Icourse,

The keywords can be tweaked later by your SEO if needed. At this point, other than the top one or two keyword phrases for that product, they should focus on providing visitors with all of the information they will need to make a purchasing decision.

I like to check with customer service to find out what questions people have been asking about various products. For example, if they tell me a lot of customers call in about whether Item A will work with Item D I will be sure to include "Works with Item D." in the product description, perhaps under an FAQ or Specs. tab on the product page. If it is a new site, or if you don't have a customer service department, Q&A sites like Yahoo Answers often provide some good ideas.

In my experience, the more detail the better as long as it is presented well. You don't want one long page of text, but if you can provide all of the specs, options, instructions and details on the product page it will not only help your rankings, but will probably save you money in call center costs.

The more instructions you give a good copywriter the worse their writing will be, in my opinion. If they are professionals (you get what you pay for with copywriting!) and are writing high quality copy about luxury products I would just let them at it in small chunks. You can have your SEO make minor adjustments and recommendations as they go.

I will leave this question open for discussion.

Nico,

Great question. Technically speaking, a permanent 301 redirect should stay up indefinitely. However, I have heard more than one Google employee say publicly that they should stay up for "at least a year". At that point I think you can take them down and the destination page will retain the link juice from the old links since Google will have supposedly updated their link graph. However, I don't trust that most Google employees really know what's going on under the hood all the time, and recommend leaving any redirects from pages with external links in place for as long as you have control over it (e.g. you obviously couldn't leave them in place across domains if you sold the old domain).

If the page doesn't have any internal or external links pointing to it you can safely remove the 301 redirect at any time. A year would be plenty.

Having looked at the site I can see that the content is more than unique and useful enough. Great job on that!

By using "rel next / prev" in this way you are giving Google the signal that you want the first page (i.e. rio-negro-amazonas-regenwald-expedition-uebersicht.php ) to rank higher than the other two pages for most searches, but that the other pages are unique and should stay indexed. If this is what you intend then I think it is a great plan. However, if all of the pages are equally important, and if each has its own search terms to target, it may be better to let the subsequent pages stand on their own.

It sounds like this is working for you at the moment. Thank you for sharing your findings with us!

Hello Nico.

I think I have some good news for you. Unless there is live link to those URLs (internal or external) there is no need to redirect them. In my experience more than 99% (an educated guess) of product pages do not have any external links. So assuming you have not cleaned up your internal linking (if you haven't, you should) there really is no need for the 301 and you can just let it resolve as a 404.

As long as the old URLs return a 404 status code they should be removed from the index within a matter of weeks. However, if they don't have any external or internal links - again, most of those pages probably do not - it may take months for Google to decide to recrawl an indexed page without any link paths to it. Googlebot couldn't just "crawl" its way there, but would have to retrieve the URL from a database just to check it, which would take longer. I have a tested and recommended solution to this issue, which can be found here on the Moz.com blog.

Thank you for sharing your direct experience with this strategy.

Do all of these URLs share the same rel canonical tag, or do they each have their own self-referencing rel canonical? I am assuming they each have their own if they are all showing up for searches.

It would really help if you could share the domain so we could have a look. However, as long as the content on each page is not "thin" and is mostly unique to that page I think this strategy would be fine.

Hello Matt,

Two things:

#1 - "The amount of pagerank that is lost through a redirect is currently the same as the amount of pagrank that dissapates through a link". So they are exactly the same. Source: http://www.youtube.com/watch?v=Filv4pP-1nw

It was thought that more pagerank was lost through a redirect because Google does not want webmasters relying on redirects instead of actually updating internal links. At least that's what Matt told us at a search conference many years before denying it in this video. So.... That's sort of a non-issue - for now. Until Matt changes his mind about how he wants to explain it again.

#2 - It is still "best practice" to update your internal links after a site migration instead of relying on the 301 redirects. The reason is that you don't just migrate once. Think long-term and you will see there is always going to be a possibility of changing the URLs again at some point in the future. If you do not update your internal links you're going to be sending users through multiple redirect hops, which will THEN probably lose more page rank than what is lost in a normal links because it will be happening multiple times.

Hello BenVer,

It really all depends on your situation. I think most SEOs these days are going for either rel next/prev or a view all canonical page (see links below). However, I still see a lot of sites doing well with a rel canonical tag on paginated pages that points to the main/first page. Rather than saying this one is "best" I feel more comfortable proving you with some resources so you can look into the best solution for YOUR needs...

Rel Next Prev
 Official Google Video about pagination with rel=“next” and rel=“prev”
Pagination with Rel Next / Prev on the Google Webmaster Central Blog

View All Canonical
Google Webmaster Blog: View All in Search Results

Rel Canonical
Five Common Mistakes with Rel Canonical on the Google Webmaster Central Blog
Rel Confused? Answers to Your Rel Canonical Questions by Dr. Pete on Moz.com

Most of the time the URL will be different for the affiliate link because tracking is involved. If this is the case then you don't need to worry about which one comes first, or one devaluing the other.

However, if both URLs in the href tags are exactly the same, the answer is Yes - the one nofollowed link could devalue the followable link, even if the nofollowed link is below the followed links. The issue Chris Menke is referring to above has to do with anchor text, not follow Vs nofollow. Yes, the anchor text in the first link is what counts. However, if any link on a page is nofollowed Google may choose not to follow that same link elsewhere in the document. This is an issue that was looked into when people were using nofollow tags to sculpt pagerank.

Hello Bastian,

With regard to outdated products the best solution I have found is to 301 redirect the old product URL to the closest category page. I prefer this to redirecting to another product page because once the new product is outdated you'll be going through multiple redirects.

Look into updating the old blog posts and making them more relevant for readers today. If you are unable to do that I would just leave them where they are. As they get outdated they should drop in the rankings anyway. Or you could 301 redirect them to a more recent post.

Hello Taylor,

You are correct that there are risks in this strategy, however I think they are mostly mitigated if you use the cross-domain rel canonical tag (i.e. a rel canonical tag with an absolute path pointing to the "homebase" page). You will also be keeping those subdomains from ranking if you use this tag, so if the goal is to have a landing page for organic search traffic optimized for each region then a rel canonical tag probably isn't the best option, as the only page to rank will be the homebase page, or whichever you say is the canonical version. Make sense?

Please let me know if you need further assistance with this question.

Hello Axello,

Are you still having this problem? If so, someone may need to look at your GWT setup to diagnose the issue. You can try some of the SEO companies and consultants listed here in order to hire an analyst to help fix the issue.

If you feel comfortable sharing the domain I can have a look for you to see if there are any answers to be had without logging into GWT.

Hello Pat,

I do not have experience merging a Ruby site with another type of site, but I think we are confusing issues here anyway. You can have content in a database that gets served up anywhere. That is, you can pull that content into ten different websites if you wanted to. The database issue is almost irrelevant to the SEO issues, which have mainly to do with loss of pagerank from URL changes, and possible duplicate content issues. A 301 redirect from the old URL to the new one would take care of both of these issues.

If you are unable to redirect all of the old content my suggestion would then be to figure out which URLs have external links and redirect those. Let all of the other ones return a 404 or 410 status code so those URLs will be removed from the index since the content will exist on the new URL and you don't want two URLs with the same content indexed simultaneously.

Please let us know if we have missunderstood the question or if we can provide more help with your original question. You may want to post your Ruby question in another thread to ensure the right people see it.

Thanks!

Hello Holger,

I apologize for the wait on this. We rely on the community to help answer questions, but sometimes nobody is able to help out in a timely manner so we answer them ourselves as well.

I do not think rel next/prev is the best solution for the situation described. I think the best practice would be to have all of that content on one page. You could change the view of the content (such as when someone clicks the "itinerary" tab) by adding a hash symbol (#) to the URL (e.g. amazon-tour/#itinerary amazon-tour/#prices) to avoid duplicate content issues and make the landing page more robust and useful. You might combine this with the use of a rel canonical tag for that page.

Please let me know if you still need assistance with this question. Again, sorry about the wait!

Sri you "should" rel nofollow advertising links. It is as simple as that. If the "advertiser" says they won't pay if it is nofollowed then they didn't want advertising, they wanted to buy a link - which is against Google's webmaster guidelines. I'm not judging one way or another or saying anything about whitehat vs grayhat vs blackhat vs anything else. All I'm saying is a link that has been paid for, and passes pagerank, is technically against Google's guidelines. If you and your clients each go into the agreement knowing this and understanding the risk then that is up to you.

I think it makes sense to do this when using the PPC channel in order to test out page design, layout, keywords, etc... However, once you find a better-converting page for PPC you should also test it out on organic traffic. Assuming you can continue to rank well for your top keywords and the page converts organic visitors better it is best to send PPC traffic to an organic page due to the quality score issue described by Ron.
Plenty of businesses have separate landing pages for their PPC traffic though. As much as I would like to say that having a high-converting page is best for both, the truth is that sometimes the highest converting page is not at all SEO-friendly in terms of content (e.g. one image and a big fat CTA button). You can often meet in the middle by incorporating drop-down content divs, jquery sliders, etc... but not in all cases.

I'm just going to answer your question directly. This was your question:

"Whilst it must surely be "best practice" for all on-page links to go directly to the 'right' page, are we harming our own interlinking and even 'page rank' by being tardy in working through them manually?"

Short Answer: As long as you are working to update those internal links, and you have 301 redirects in place during the meantime, you should be fine.

Technically speaking, it is best practice to link directly to the page internally, rather than relying on 301 redirects. Yes, it is true that a very small (very, VERY small so as to be virtually undetectable) amount of pagerank is lost when redirecting, it only becomes an issue when you begin adding redirect on top of redirect. Keeping your house clean, so-to-speak, by not relying on redirects to fix your broken internal links will keep this from happening, and is exactly what the tiny amount of pagerank loss is said to be created for (to discourage webmasters from relying on redirects to fix broken internal links) - if you believe Matt Cutts.

With that said, you may indeed have many other issues to deal with, as do most sites that have a geotargeted, deep URL structure like the one you have outlined. Panda slammed a lot of sites like that pretty hard. But all of that is beyond the scope of this question.

I hope you find whatever is wrong and get your traffic back. Good luck!

Someone could probably have an answer to you within minutes if they had the domain URL available.

RE: Competitive sabotage, I very highly doubt it.

RE: Having just occurred - That is often a sticking-point for no good reason. Do not be concerned so much as to why it wasn't an issue before and focus on how to fix it now. Google's algorithm changes all the time. Your standing in the algorithm changes all the time. Trust can be lost if you get a penalty, even if you get out of it. One external link too many going to https, or one change in the crawl path so Googlebog ends up on the https site via a relative path link... Things can suddenly change for a variety of reasons. However, if you do what is being suggested you are very likely to put this issue behind you.

Here is what I do with eCommerce sites, typically:

• Rel canonical both versions to the http version
• Add a robots.txt block and robots meta noindex tag to shopping cart pages
• Use absolute paths, if possible (e.g. http://www.domain.com/file/ instead of .../file/), especially in your primary navigation and footer links.

If that doesn't work please let us know and we can evaluate the site for you.

Good luck!

Hello Pikka,

The paginated pages each have a self-referencing rel canonical tag and content that is duplicate to the main page. I think this could be your problem.

Example:
http://www.bookatable.com/uk/101917/oxo-tower
Rel canonical = http://www.bookatable.com/uk/101917/oxo-tower

http://www.bookatable.com/uk/101917/oxo-tower?reviewPage=4 (same content as above)
Rel canonical = http://www.bookatable.com/uk/101917/oxo-tower?reviewPage=4

We could make this as complicated as can be, but I'd be willing to bet money that if you simply made the rel canonical tag for the paginated pages the same as the canonical page (e.g. http://www.bookatable.com/uk/101917/oxo-tower) you would see some lift in rankings and traffic.

Good luck and let us know how it goes!

That sounds like a winning plan Dr. Pete, though I'd append 2.1 "Request removal of directory in Bing and Google webmaster tools".

Hello John,

This is a very good question, and something people don't often think about when blocking the navigational paths on their site from being crawled.

Depending on how fast your category pages load and how many products are on each of them, you may consider a View All Canonical page: http://googlewebmastercentral.blogspot.com/2011/09/view-all-in-search-results.html

There are many different ways to handle faceted navigation problems, including javascrpt, GWT parameter handling, robots meta, robots.txt, rel canonical... and combinations of these. The right approach should be customized for your specific needs. When possible, I prefer to allow Google to crawl and index down to a certain level of faceting, similar to allowing them into sub-categories (though it depends entirely on your taxonomy) but not tertiary (i.e. sub-sub) categories. For the next couple of levels I might allow them to crawl, but not index. And once it gets down to 4 or 5 levels deep (e.g. /?category=1&size=5&color=blue&price=low&this=that&so-on=so-forth...) I just block them from being both indexed and crawled (i.e. Meta NOINDEX,NOFOLLOW or robots.txt block) to save crawl budget by avoiding spider traps.

With all of that said, if you are giving Google an XML sitemap that contains the indexable URLs to all of your products they should have no problem indexing them, regardless of whether or not they can crawl all the way through your faceted navigation.

Disallow: /photodisplay.asp?*

That should do it. But just to be safe you can add another one for:

Disallow: /photodisplay.asp

There is very, very, very, very little danger of you blocking your entire site from being crawled if you add those disallow statements to your robots.txt file. If you're an SEO your job is to "mess with" the robots.txt file. Furthermore, trying to dynamically change the robots meta tag to noindex based on page-type is going to be much more tricky and potentially dangerous than adding a line to the robots.txt file.

Don't forget to remove the pages from the index using the URL removal tool in GWT once the block has been added.

Also I'd stop linking to those pages. It is best practice not to link to pages that you don't want indexed if you can help it. I'd go the lightbox route you mentioned above. This is something I do on my Wordpress sites too.

Good luck!

I agree with Dr. Pete here, though I think the easiest solution would be to simply block the entire /design-your-own-jewelry/* directory from being indexed using robots.txt and, to Dr. Pete's point, you'll want to remove that directory from the index in both Bing and Google webmaster tools, as discussed here:

http://googlewebmastercentral.blogspot.com/2007/04/requesting-removal-of-content-from-our.html (see the section under "entire directory")

Something I think about with regard to robots.txt block Vs meta robots block is crawl budget. Google has to access a page to see the meta noindex tag, while a single disallow statement in the robots.txt file can save Googlebot the hassle of visiting potentially thousands of unnecessary pages.

If down the road you figure out a way to put custom content on those pages and want to try and rank for things like "Custom Garnet Pearl Bracelet" or "Design Your Own Beaded Bracelet" then I'd look into some of the other options discussed here. Until then I feel they would just be complicating something as simple as the need to remove very thin, mostly duplicate content from the index.

While the blog format of showing a snippet with "read more" links is commonplace and perfectly acceptable, I would argue that it DOES matter whether the blog is on your domain or not. It would be better to have it on your domain. You don't want to seem like you're trying to beef up the content on your product and category pages by simply pulling in someone else's RSS feed snippets.

The longer you wait to move your blog over to your main domain the more difficult it will be. I would do this sooner rather than later.

It seems the general consensus is to launch the "good enough" site without blocking Google, and to fix the SEO issues as soon as possible.

However, I'd say that it really all depends on what those SEO issues are. For example, if you think you're going to be releasing thousands of non-canonical URLs into the SERPs without using any "fixes" it could be a long time before you get those out of the index once they're "fixed", especially on a new site with no deep external links. If waiting a couple of weeks before allowing the site to be indexed could save me from having to do thousands of individual redirects (as in those not handled easily by regular expressions), and could keep my site from launching with thousands of pages of thin and near duplicate content (why not start off in Google's good graces? Why start off on the wrong foot?) I would seriously consider blocking everything but the home page in the robots.txt file.

You would want the home page to be indexed no matter what because the launch will likely coincide with lots of press, advertising, etc... and people will be searching for your domain and/or brand. This would allow the "domain" to be indexed, which would take care of the date of indexation ranking factor discussed above (though in the grand scheme of things a few weeks is not going to matter), and would allow you to show up for a large proportion of searches (i.e. brand and navigational queries) since you would be unlikely to rank for many big non-brand searches out of the box anyway.

Then again, if you are just concerned with some small SEO issues, such as adding alt attributes or improving internal linking, I'd go ahead and launch.

Samuel, I'd like to start over fresh in answering your question because there are too many thoughts going on here at once. Let's start with the obvious: Can you provide a link so we can have a look at the content you wrote? It is possible that you have crossed some sort of content overoptimization threshold if it seems to be keyword-stuffed.

Regarding timing, Google has a lot of databases and they don't all update simultaneously. It isn't uncommon for the title tag to update in the SERPs before the page content updates in the cached snapshot. Furthermore, algorithmic calculations, including whether or not to apply a filter or penalty, do not always happen at the same speed that indexation happens.

All of this is to say that just because a drop in traffic happened after you updated content doesn't mean the content was the cause of that drop. Post hoc ergo propter hoc (after this, therefore because of this) is a logical fallacy. But there does seem to be a strong temporal correlation so it is the obvious place to start ---> Can you share a link to the content?

Trenton this is a great question. Yes, you should go ahead and redirect that page to the new home page, as it was your former home page. Even though the content is different you should be able to move the pagerank over to the new home page by use of the 301 redirect. Google does treat some redirects to the home page as a 404, such as when someone redirects every page from one domain to the homepage of another, but I think you will be fine in this situation.

Hello Federico,

There are enough tag pages still indexed that I'd go ahead and remove that entire directory in Google Webmaster Tools. You may consider a noindex tag on archive pages as well.

In regard to your original question, I don't think it would necessarily remove an algorithmic penalty, but it would be good for your brand in the long run. If you're going to do a change like that you might as well do it while traffic is low anyway.

Yes, 301 redirect the .net site to the .com site on a page-by-page basis.

I would also look into creating a few more useful non-blog pages on the site. Some type of innovative resource that advertisers and/or publishers would find super useful would help you build up the authority of the main site, which would be important considering how relatively few pages and links you have there compared to the blog. An inforgraphic wouldn't cut the mustard, but I'm sure you can come up with something since it is a very well-done site visually.