Welcome to the Q&A Forum

Google can read PDFs, and returns them in search results, but some users might prefer to view an HTML version.  Also, it looks like images in PDFs are not indexed, according to the 2nd post below.

Regarding duplicate content, Google says (2nd post below):

Q: Is it considered duplicate content if I have a copy of my pages in both HTML and PDF?
A: Whenever possible, we recommend serving a single copy of your content. If this isn’t possible, make sure you indicate your preferred version by, for example, including the preferred URL in your Sitemap or by specifying the canonical version in the HTML or in the HTTP headers of the PDF resource. For more tips, read our Help Center article about canonicalization.

These will be of interest to you:

http://www.google.com/support/forum/p/Webmasters/thread?tid=4472512a5515686b&hl=en&fid=4472512a5515686b00047d6de91c24fa&hltp=2

http://googlewebmastercentral.blogspot.com/2011/09/pdfs-in-google-search-results.html

As far as I know, you can't do this in Google Webmaster tools.  It would have to be a parameter like shopby= instead of shopby/ as you suspect.

This sounds like a good place to use the rel=canonical tag.  You want those pages to exist, and if people link to them, you want them to pass their link juice back to the /tshirts/ page.  These shopby pages will have a subset of the info on the /tshirts/ page, so Google should respect the canonical tag.

Other search engines beyond Google should respect the canonical tag as well, so that's another benefit to doing it outside of Google Webmaster Tools.

Here's some reading:

rel=canonical: http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html

Using canonical to a view all page: http://googlewebmastercentral.blogspot.com/2011/09/view-all-in-search-results.html

Google just announced some tags to help support pagination better.  They say if you have a view all option that doesn't take too long to load, searchers generally prefer that, so you can rel=canonical to that page.  However, if you don't have a view all page, then you can put these nifty rel="next" and rel="prev" tags in to let Google know your page has pagination, and where the next and previous pages are.

It sounds like you don't want to go the view all route, so you should read the second post below and you can implement the rel="next" and rel="prev" tags.

(from the view all post below) However, if you strongly desire your view-all page not to appear in search results: 1) make sure the component pages in the series don’t include rel=”canonical” to the view-all page, and 2) mark the view-all page as “noindex” using any of the standard methods.

View all: http://googlewebmastercentral.blo...

next/prev: http://googlewebmastercentral.blo...

Google just announced some tags to help support pagination better.  They say if you have a view all option that doesn't take too long to load, searchers generally prefer that, so you can rel=canonical to that page from your series pages.  However, if you don't have a view all page, then you can put these nifty rel="next" and rel="prev" tags in to let Google know your page has pagination, and where the next and previous pages are.

View all: http://googlewebmastercentral.blogspot.com/2011/09/view-all-in-search-results.html

next/prev: http://googlewebmastercentral.blogspot.com/2011/09/pagination-with-relnext-and-relprev.html

Google just announced some tags to help support pagination better.  They say if you have a view all option that doesn't take too long to load, searchers generally prefer that, so you can rel=canonical to that page.  However, if you don't have a view all page, then you can put these nifty rel="next" and rel="prev" tags in to let Google know your page has pagination, and where the next and previous pages are.

View all: http://googlewebmastercentral.blogspot.com/2011/09/view-all-in-search-results.html

next/prev: http://googlewebmastercentral.blogspot.com/2011/09/pagination-with-relnext-and-relprev.html

A bot?  You're scaring me a bit, like the lunch lady in Billy Madison.  Are you going to write a bot to spam your links on their comment pages?  That sounds fruitless... as these comment pages are already filled from top to bottom with spam.

If you use Firebug in Firefox, or Chrome, you can write what text you want in the field and add the link, then inspect the input where you see the text, and grab all the HTML that was generated by adding the text.  Then it's just a matter of having a bot inject that text into the page.  You'll still have to beat their captcha though to fully automate the process.

I tried it and it worked...

My guess is you're not putting the "http://" in the Link URL field, so the comment thinks it's a relative URL, so it ends up amending whatever you put after the page URL in the link text.  Make sure to put the absolute URL in the Link URL field, starting with "http://", and it should work fine.

You can do it the same way you can add links in for comments in this Q&A forum.  Write the anchor text in the comment, then select the text, click on the link button, and then put the anchor text in the Link URL field in the pop-up that appears.

link.gif

Hey Antonio!

It looks like all the elements on each page are in the source when the page loads... so I'm not sure where AJAX is being used.  There's a lot of JavaScript/CSS effects, but I don't see anything where JavaScript is loading something after the initial page load.

Usually the best thing to do is use subdirectories on one domain, to consolidate your domain authority/strength.  Here's some reading if you want to learn more:

• A recent Q&A about it with a staff endorsed answer: http://www.seomoz.org/q/subdomains-vs-subfolders-for-unique-categories-topics
• An SEOMoz blog post about it from 2009: http://www.seomoz.org/blog/understanding-root-domains-subdomains-vs-subfolders-microsites
• A presentation mostly pertaining to these choices for international sub-sites, but still relevant: http://www.distilled.net/blog/seo/mozcon-international-seo/

It can be done.  Make sure your development team reads through http://www.seomoz.org/blog/how-to-allow-google-to-crawl-ajax-content and http://code.google.com/web/ajaxcrawling/ before coming to any conclusions.  If you have an AJAX based website, you'll have a little extra work to do depending on the route you take.

Since you have no control over those other websites, you can't really do anything to get them to remove the links.  But because of this, incoming links to your site will never penalize your site.  If they did, you're right, a competitor could get sites penalized on purpose and point lots of links from them to your site.

What else could have caused your site to get penalized?  Could one of the Panda updates have caused it?  Check out the time line here and see if it lines up to when your site lost its traffic: http://www.seomoz.org/google-algorithm-change.

Definitely 301 redirecting them is the way to go.  Make sure each page on the redirected domain maps to the corresponding page on your main domain.

It sounds like you know a few pages on these domains you want to redirect that rank well.  I'd pick one of these domains to start with, and redirect all of its pages to the corresponding pages on your main domain.  Wait until Google reindexes those pages that rank well, and try those searches again.  That page on your main domain should rank at least as well as the page on the old domain, and hopefully even better since their link juices are consolidated.

If you're highly risk-averse, you can continue doing this domain by domain.  Otherwise, you can batch them and do several at once, or just redirect the rest of them.  Just make a note of the searches where pages on these other domains were doing well, and check that those pages on your main domain does just as well after those pages get reindexed.  Personally, if it worked for one, I'd feel pretty safe doing all of them, and then keep checking the SERPs afterward to make sure nothing bad happened.

How gradual the process is will be determined by how quickly the pages are reindexed.  If they're reindexed quickly, you'll be able to see if it's working and can keep moving forward.

rel canonical is for letting search engines know that a page on your site has duplicate content of another page.  However, I doubt all the pages on your site are duplicates of your home page, so the most likely outcome is that the search engines would ignore your rel canonical tags and just index the pages as normal if they're not similar enough.

This is not really the intent of the tag, and to me it sounds like it falls close to the black-hat side of things.  If you really want the rankings from those pages, and don't care about them, you can 301 redirect them to your home page like Joshua suggested.

Here's the SEOMoz post from about a year ago about rel canonical.  It also links to some good resources: http://www.seomoz.org/blog/complete-guide-to-rel-canonical-how-to-and-why-not

I think it's worth it as it should only take a few minutes to set up, and it's good to have a robots.txt, even if it's allowing everything.  Put a text file named "robots.txt" in your root directory with:

<code>User-agent: *
Disallow: 

Sitemap: http://www.yourdomain.com/none-standard-location/sitemap.xml</code> 

Read more about robots.txt here: http://www.seomoz.org/learn-seo/robotstxt.

Somehow Google is finding these pages, but you're disallowing the Googlebot from reading the page, so it doesn't know anything about the meta noindex tag on the page.  If you have meta noindex tags on all of these pages, you can remove that line in your robots.txt preventing bots from reading these pages, and as Google crawls these pages, they should remove them from their SERPs.

Here's a good SEOMoz post about this: http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts.  What's most likely happening is that the disallow in robots.txt is preventing the bots from indexing the page, so they're not going to find the meta noindex tag.  If people link to one of these pages externally, the disallow in robots.txt does not prevent the page from appearing in search results.

The robots.txt syntax you're using now looks correct to me for what you're trying to do.

This is what rel canonical tags are for!  Read more about them here: http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html

It sounds like the aspx pages are still being served?  If so, you should 301 redirect them to the new friendly URLs with the htm extensions.  Google doesn't care about which page extension you use.

I'm not sure about the answer to the first question, but there was an SEOMoz post awhile back about scraping Facebook comments, so that you could get search engines to be able to read them.  If you can scrape them, then you should be able to set something up to detect with some frequency when new comments are added.  If you use them, you'll probably want to set this up so the comments can be indexed.

I've never used Facebook comments, so there might already be a way for Facebook to notify you when comments are added.

Add canonical tags to all the https pages that have a http counterpart pointing to the http version.  Here's the original Google Webmaster Blog post about canonical tags: http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html

Your meta descriptions aren't very long at 60 characters.  You could write longer meta descriptions that include more info specific to that watch.  Like, for http://www.traxnyc.com/Mens-Diamond-Watches-G-Shock-Watches-Casio-G-shock-Atomic-Solar-Watch-GWM5600-1V-item25463.html, the meta description is "Mens G-Shock Watches White Stainless Steel Item Price : $120", but the product is "Casio G-shock Atomic Solar Watch GWM5600-1V".  When this page appears in the SERPs (when I search for the page title at least), the meta description is short so Google fills in a lot:

Mens G-Shock Watches White Stainless Steel Item Price : $120. ... TraxNYC is a leading retailer of Diamond Jewelry in the heart of New York's famous Diamond District .... Full Size Picture for Casio G-shock Atomic Solar Watch GWM5600-1V ...

Each subdomain may have its own robots.txt file.  So for that subdomain, you can put:

User-agent: *
Disallow: /

In the robots.txt, and that should do it.

Please note that disallowing pages in robots.txt will not necessarily mean they won't appear on search result pages.... if people link to pages that are disallowed on that subdomain, they can still appear in SERPs.  I had this happen with a few pages, which leads to funny listings in the SERPs because Google has to guess what the page title and description of the page should be, since it's not allowed to read the page.  The meta noindex tag is the way to go if you want to be really sure the page doesn't appear in the SERPs.  If you use that, don't disallow the page.  Here's a recent SEOMoz post about it: http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts

If you just want the directory page removed from SERPs, but not the images, you can put a meta noindex tag in the of the page (see here).  It'll look like .  Provided you link to the images from other pages, you should be in business.

I don't think it would hurt your rankings at all.  I would exclude it as you don't want people wandering in to your site on a directory page.

For Google, you can go into your Webmaster Tools, and go to the Site configuration > URL parameters page.  You can set the parameters in your URLs to things like "Sorts", "Paginates", or "Narrows", so that Google will know how the parameter changes the info on the page.  Once Google knows how the parameter effects the page, presumably they'll give the right links and display URLs in their search results.

One thing you can do is install Google Webmaster Tools on your site, log in and go to Site configuration > Settings, and set your Geographic target to the United States.

For Bing and other search engines, you can set the content-language in a meta tag or in your HTTP headers (see here).

If my detective skills aren't lacking, his site is http://www.njtraininggrounds.com/.

Using display:none is fine.  There's JavaScript on the page that will make those items display when the users takes some action on the menu.  Google has some way of reading the JavaScript to see what may be displayed, and at this point, I'm sure they're very good at detecting these types of menus.  I don't think you can't make a menu like that without using it, and tons of sites use them.

Obviously, you shouldn't use display:none to cloak text on the page, when no user action can get that text to be displayed.

This is why the canonical tag was invented, to solve duplicate content issues when URL parameters are involved.  Set a canonical tag on all these pages to point towards the version of the page you want to appear in search results.  As long as the pages are identical, or close to it, the search engines (most likely) will respect the canonical tag, and pass along the duplicate versions link juice to the page you're pointing to.

Here's some info: http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html.  If you Google "canonical tag", you'll find lots more!

I like lowercase because when I type URLs by hand, I don't think to capitalize things.  If you capitalize things, you have to get the casing right to make the URL valid (unless you're setting up all sorts of fancy redirects), otherwise you a 404 and are left scratching your head.  Also, I agree with Dan that it looks better.

Hyphens vs. underscores is a classic question; Matt Cutts says to go with hyphens: http://www.youtube.com/watch?v=Q3SFVfDIS5k.  I like that better too.

For images at least, they say alt >> title.

As a user, when I hover over those links, I'm going to think it's strange and annoying when that title text keeps appearing for every link.  I doubt you're going to be fooling Google with that.  It's blatant keyword stuffing.  As for whether it's worth penalizing, I don' t know.  I don't think Google takes the title attribute into account very much for rankings anyway...

I'd get rid of it them, or make the title text match the links it appears for.

One thing in addition to Ryan's answer, rather than implementing language meta tags, you can set the content-language in your HTTP headers, according to Bing: http://www.bing.com/community/site_blogs/b/webmaster/archive/2011/03/01/how-to-tell-bing-your-website-s-country-and-language.aspx.

I'm not sure if that's supported beyond Bing though.

Placing canonical tags isn't an option?  Detect that the page is being viewed through the subdomain, and if so, write the canonical tag on the page back to the root domain?

Or, just place a canonical tag on every page pointing back to the root domain (so the subdomain and root domain pages would both have them).  Apparently, it's ok to have a canonical tag on a page pointing to itself.  I haven't tried this, but if Matt Cutts says it's ok...

A few things I noticed... on my way out of the office so this is all I got:

• Some canonicalization issues.  Both /search-engine-optimisation/best-seo-tools/ (trailing slash) and /search-engine-optimisation/best-seo-tools (no trailing slash) serve pages.  It looks like you go with trailing slashes, so the version without should 301 redirect to the version with.  Also with your home page, you use http://www.onlinemediadirect.co.uk/, but http://www.onlinemediadirect.co.uk/index.php also returns your home page.
• Your 404 page is kind of unfriendly.  It's better when you return your 404 page within the shell of your site, so it's easy to get back to where you meant to go.

Non-SEO stuff:

• The compass on your home page looks like Apple's Safari compass to me (http://www.apple.com/safari/).  That's the first thing I thought of when I saw the home page.
• I don't care for the dark text, and the animation/delay on the highlight of your dropdown menus.  The text is a bit hard for me to read, and the delay feels strange.

It's the new form of what we like to call linkbait.  People love infographics, and they like to be the first to discover new ones and share them with their friends.  Then everyone tweets and retweets about them, posts Facebook statuses about them, links to them from their blogs, and before you know it, it's gone viral.

http://dailyinfographic.com/ is a site with just infographics.  Usually they contain a lot of related data, charts, and graphs, laid out in an eye-pleasing and engaging way, all in one gigantic image.

People may have been tweeting about only some of the pages, or linked to some of the pages, or maybe the bot just randomly decided to index some of them and not others?

If you create individual sites for each location, this will automatically geotarget your sites in the SERPs for those countries.  Each site will be starting from scratch in terms of domain authority, but if you can overcome that hurdle you'll be off and running.

If your original domain was a .com, it seems more normal to have the directories for different languages, but with a .co.uk would that be weird?  I'm an American and .co.uk sites don't pop up on my radar all that often.  Doing it the way that NicoDavila has pointed out does have the bonus of providing your current domain authority to each localization of the site.  Then you can set those to geotarget those country specific directories in your site in Google Webmaster Tools, and by setting the language & country in a meta tag or in your HTTP headers (for Bing & other search engines).

A few things I noticed:

• Almost all your pages try to get /skin/frontend/blank/default/css/menu.css once and /js/livechat/js.js twice, and these files don't exist and return a 404.
• Are you allowed to change that 404 page to something more suited to your site, with your header and footer?  If so, I'd change it from the hostgator default.
• #5 could be quite an undertaking... positioning the text just right, controlling for wrapping in different browsers. I'd start by adding alt text to the images first, so Google can read that and know what the image is supposed to be of.

For Google's opinion, you'll want to watch http://www.youtube.com/watch?v=mY9h3G8Lv4k, and read http://googlewebmastercentral.blogspot.com/2009/11/help-google-index-your-mobile-site.html.

Add it to your site.  You want people to know the blog is part of the site, and you want people to be able to get from your site to the blog and vice versa easily.  Also, you want your site's rankings to benefit from the traffic you bring in via the blog, and vice versa.

To make it be treated as part of your site, you should set it up under a URL like mysite.com/blog, vs. blog.mysite.com.  The subdomain approach will get your blog treated like a new site.

If you think people may link to these landing pages, you'd be better off putting them on your main domain, so these pages help contribute to your main domains rank.  Generally it's better to put quality pages on your domain, instead of putting landing pages on these extra domains, and pointing them to your site.  It doesn't sound like your extra domains will have much pagerank to pass to your main domain.

Also, more importantly (for me at least), it would be strange to arrive at a landing page, and click a nav link, and arrive at a completely different domain that uses the same nav/header/footer.  I'd feel like there was some spoofing going on...

I'd just set up these extra domains to 301 redirect to my main domain, and leave it at that.

Ryan, spot on as always.

One other thing, it sounds like some of the canonicals you're placing on pages would be better suited to 301 redirects, like correcting a URL for not having a trailing slash or not.  If you can avoid using canonicals and use 301 redirects instead, that's the preferred method for resolving duplicate content issues.  Canonicals are more for when there are parameters on the URLs, and you can't get away from serving the pages with those parameters.

You shouldn't use nofollow for any links on your site to pages on your site, unless you really don't want that link followed.  Even in that case, you should meta noindex that page, or disallow it in your robots.txt rather than nofollow the link.

You used to be able to do some pagerank sculpting with nofollow, but they changed it a few years ago so that nofollow burns pagerank, rather than sculpt it.  For example, suppose you have page A has 6 link juice "votes" to pass, and links to 3 pages:

• page B
• page C
• page D

If you follow all the links, each page gets 2 votes.  If you nofollowed the link to page D, then page D will get no votes, but page B and C still get only 2 votes each.  So there's no reason to burn those votes to page D, right?

Back in the day in this example, page B and C would each get one of the extra votes that D didn't get (so, 3 votes each), but I guess Google didn't like people over-sculpting their sites to try to push just a few of their pages to #1, and ignoring the rest.

I don't have an answer to your question, but if you're looking for some more reading about subdomains vs. TLDs, here is a presentation given at MozCon: http://www.distilled.net/blog/seo/mozcon-international-seo/. The slideshow has some info about it, and a bunch of other good stuff.

Google doesn't frown on them, but I would really question their value at this point.  You could probably put your money somewhere more useful.  I really like EGOLs answer to http://www.seomoz.org/q/how-would-you-use-500-to-improve-your-site-s-visibility.

If you do want to buy a link, a good one is the Better Business Bureau.  Here's a post about it.  You can get a followed high PR link that way.  There's also the SEOMoz directory list.