On a sidenote, if you want to prevent indexation of them you can probably add the URLs in to Robots.txt or disallow a specific folder for those images.

On the flipside, if you want to get them indexed, you can manually link to the images and submit an image sitemap to google with just those particular images. Doing both is better than just one.

Quick anecdotal reference - for the 1 site I just checked (which allows CSS to be crawled), no they are not indexed.

However, Google crawls CSS and can load those images when you Fetch And Render inside of GWT I believe, so theoretically they can index them.

Next question is what would cause them to do so (eg inbound links to that image file), but I don't have an answer for that.

That's the route I'd push for as well I think.

Agreed on experimentation. Please report back if you get a chance to test this. Perhaps choose a small number of PDFs on this site redesign and leave the link off of them?

I haven't seen any studies with <embed> the way I have with <iframe>. <embed> is also used for video and flash, but neither would be indexed the same way as PDF so hard to compare. The embed tag is pretty standardized, so I really doubt they wouldn't crawl this similarly.</p> <p>IIRC in the ugly era of flash, it was proper to have a <noscript> {crawlable content here} </noscript> section after the <embed>, so that's one comparable situation, but that's due to the flash itself not being crawled well.</p> <p>If it's not a hassle, I would add the text link to the PDF that says "download full PDF" or similar. If it is a hassle and takes longer than a couple hours, then it's a harder call.</p> <p>Similar thread that could be helpful:</p> <ul> <li><a href="http://stackoverflow.com/questions/3686331/does-google-index-html-content-supplied-by-the-object-tag">http://stackoverflow.com/questions/3686331/does-google-index-html-content-supplied-by-the-object-tag</a></li> </ul></iframe>

Hey Noah,

Ray's comment above is correct but might be unclear. Yoast handles this the way you want - you've got a theme issue. This code is wrong:

if(!wp_title("",false)) { echo bloginfo( 'title');} ?>

This tells Wordpress to output "page title - blog name" as the H1. Yoast isn't interfering in that. You'll want your H1 to say this:

As a 2014 follow up to anyone reading this thread, Google later released a tag labeled "x-default" that should make the self-referencing canonical question moot.

Read more at http://googlewebmastercentral.blogspot.com/2013/04/x-default-hreflang-for-international-pages.html

If this is still an issue you're facing, have you checked the sitemap settings to see which page types are getting included? For example, a site with a few thousand tags that are not entered in the sitemap but not yet set to noindex could easily produce extra pages like this.

The next step is parameterization. Anything going on there with search URLs or product URLs? eg ?refid=1235134&q=search+term or ?prod=152134&variant=blue

If you really want to scrape through Google, get a list of your sitemap and scrape queries like "inurl:domain.com/a", "inurl:domain.com/b", "inurl:domain.com/c". etc. This should allow you to dive deeper into the site map to see what Google really has indexed. For URL subfolders with tons of URLs like domain.com/product/a, you'll want to do the same thing at a subfolder level instead of root URLs.

You can do that with a tool like Scrapebox or Outwit. Go slow, or else you'll need to use proxies to get Google to respond fast enough. As another commenter mentioned, it's probably against TOS.

Just to confirm mosquitohawk's comments, there's not a great way to do this other than sorting through the spreadsheet.

Hopefully URLs have distinct enough subfolders that you can break them out into sections easily.