Just came across a terrific resource that reminded me you'd asked about further reading, Ben.

Check out BrowserDiet for a huge collection of resources about tuning front-end performance of websites. (You'll see #6 talks about exactly your original question)

I can also recommend reading Steve Souder's two books - High Performance Websites and Even Faster Websites - both from O'Reilly. Souders is pretty much the leading specialist in this area. He's the creator of YSlow, one of the primary tools for measuring/analyzing site speed, and is now Head Performance Engineer at Google. His website is SteveSouders.com

That'll be more than enough to get you started. Lemme know if you're still hungry for more!

Paul

P.S. The report details from tests at webpagetest.org can also teach you a huge amount, and there's a forum there run by Patrick Meenan (who built webpagetest) which is just excellent. Patrick frequently answers questions personally.

You can actually get the best of both worlds using the Redirection plugin Iain mentions, Himanshu. The plugin can be configured so that the redirects you create within it get written to the .htaccess file.

Having the redirects in the .htaccess file is preferred, because the .htaccess is processed faster and with lower load by your server than the redirects within WordPress. This is obviously much more important if you have a large number or redirects to write, as opposed to a dozen or so to deal with a few changed pages.

Paul

Taswir, every page on your website has a meta-no-index. no-follow tag in the header. This explicitly tells the search engines NOT to include you pages/posts in the search results.

On WordPress sites, this is usually caused by having the "Discourage search engines from indexing this site" box in the Reading section of the Settings checked. This is often set this way during development, then folks forget to remove it when the site goes live.

Remove the check mark from that box and the engines should start indexing your pages within a day or so.

If that box isn't checked, check for plugins that might have the same settings - like SEO plugins.

That should do it.

Paul

[Included a screenshot to show where in the navigation to find the setting to change]

03d13ae1bc

Mememax, you're talking about a completely different situation.

Screaming Frog/Xenu are only capable of broken links that your own site links to.

The SEOMoz and WMT crawls are finding missing pages that other sites are linking to on your site. I.e. broken incoming links as opposed to broken internal links.

While both are good to check, if you aren't watching for and fixing your 404s that other sites are linking to, you are potentially throwing away hugely valuable incoming link juice. (Hint: it's even more valuable juice than what is passing from your internal links)

A simple example of this occurs when you just move a page to a new URL, and update your own site's navigation. A Screaming Frog crawl would show no 404 as a result (because you updated your own site's link), but all the incoming links to that moved page could now be 404ing. It's entirely possible that other sites were linking to that page. If you don't watch your broken incoming links reports either in SEOMoz to WMT, you won't know that those other sites are still trying to link. And then you'd be throwing away that incoming link equity.

Makes sense?

Paul

P.S, Of course, you can also monitor your 404s in Google Analytics, but that only shows you broken incoming links that are getting traffic. The other tools mentioned will show you nearly all the broken incoming links even if they're not currently driving traffic. By fixing them, you'll rescue the ranking value and also be ready if they later start to drive actual referral traffic.

In my experience, Carolyn, links between subdomain and primary get the worst of both worlds.

The search engines can clearly see that two sites are interrelated  so the links between them aren't entirely "natural". Therefore I don't  find the links carry the full amount of their influence across (not to mention the subdomain is usually far lower authority than the primary.)

On the other hand, the search engines still seem to consider subdomains as stand-alone sites which must earn their authority on their own, so they don't directly benefit from the halo of authority of the primary site.

See what I mean by  "the worst of both worlds"?

The blog post Thomas refers to is quite specifically only related to how links between subdomains and primary are reported in Google Webmaster Tools, so I'd be careful about extrapolating it too far into ranking factors. (Though in essence I agree with him that these types of links seem to function more like internal than external as far as value is concerned)

As far as the location of the links on the subdomain's page, typically links in menus & sidebars (especially if they are site-wide menus) contribute a smaller percentage of whatever influence they have than if they were a natural part of the content.

And from a user's perspective (which I prefer to think of first, before SEO considerations) it probably makes more sense to include the link to related info right in the article rather than making them go looking for it elsewhere on the page after finishing the article.

You've asked a good question, and it's answer really starts to get into the "art" part of SEO as few pure data points are viable to use when answering. Which will be why you will get a range of perspectives in the answers.

Hope that helps?

Paul

[Edited to add: If the microsite is so closely tied to the topics and purpose of the main site, it may be worth considering moving it to a subdirectory of the primary so the two sets of content can better support each other's rankings.]

The main reason PageSpeed and YSlow recommend including width and height for images is as much the perception of page speed as the actual load time, Ben.

When you include the image dimensions, the browser can draw out the "containers" that will hold the images, reserving the space for them while they download. The browser can then go on the paint the rest of the pages CSS and objects around those "containers" without having to go back and redraw the whole page once the images have downloaded and their sizes are then known.

This gives the user the illusion of a much faster, cleaner page load, and hence the impression of a faster website.

Does that make sense?

Paul

[Edited to add: You should still keep doing what you're doing to produce "size-as" images for your pages. You don't want to be resizing images with the html dimensions, just reporting in html the actual size of the image]

Wow - looks like folks kinda missed the part where you asked to keep it simple, Ve

Because you site is built using WordPress software, it's actually simple and straightforward to do what you're asking. All it requires is the installation of a small plugin called "Redirection" to your website to automatically handle what needs to be done anytime you wish to change a page or post's permalink. (just to be sure - this is all assuming you're talking about changing the permalink right on the edit page for a post, not changing the permalink structure on the settings page for your whole blog)

In case you haven't installed plugins before, here's the step-by-step process. Don't be intimidated - it only looks long because I've made every step it's own bullet point for clarity

• log into your WordPress as if you were going to create a new post, but instead, look for the button called Plugins on the left sidebar
• hover your cursor over the Plugins button, and click on the Add New link that shows up
• on the new page, type Redirection into the search box and click the Search Plugins button beside it
• the plugin you want should be right at the top of the list - called Redirection (no other words) and in the description you'll see it is written by John Godley
• click the link for Install Now located under the name of the plugin
• you'll be taken to a page that tells you it's installing the plugin. (If WordPress gives you empty boxes to fill ion this new page, post back here. Depending on your site's configuration, sometimes WordPress needs a little help at this point.)
• Once WordPress has done it's thing, you should see a successful install notification on that page.
• click on the Activate Plugin link that now appears at the bottom of the "successfully installed" page you're on
• WordPress will again do it's thing and then take you to the page listing all your installed plugins. The plugin is now installed and activated!
• now go the post or page you want to change, and edit its permalink, republishing the post when you're done.
• go to your live site and click on one of the links to the page you just changed, to confirm that you end up on the correct new page at it's new address
• DONE! Go have a glass of wine (or other celebratory beverage.)

The background (if you want to know the "Why" as well as the "how")
When you change the permalink of a page in WordPress, you are changing its actual web address (also know as its URL). Even though you know you've done this, the rest of the web (and even your own website) don't know about the change so they're still looking for that page at it's old address.

When your site's server gets a request to display that page, it can't find it so it sends back a 404 error, which basically says "there's no page at the address you're asking for and I have no idea where it might be".

To fix this, you need to teach the server where it should send everybody who's looking for the old address. You do this by writing what's called a 301 Redirect into the server so that when it gets a request for that old address, it can say "there's no page at that old address anymore - it's now moved permanently to this new address and I'm going to automatically send you there."

This all happens instantaneously in the background so the visitor never sees it - they just end up on the new address. In addition, this redirect tells the search engines to give the ranking value of the old address to the new address so it doesn't look like a brand new page starting from scratch. (helps keep your page showing up well in the search results)

The Redirection plugin automatically notices whenever you change the permalink of a page and automatically writes the code necessary to give the server the 301 redirect it needs, instead of you having to muck about with all the coding the other answers were talking about. YAY for WordPress! (Still, best to change the permalinks as seldom as possible though)

The method I gave you installs the plugin directly from the WordPress parent site , meaning they've checked and approved it so you know it's safe. The plugin can also do many other powerful things, but those are for another post

Do let me know if anything's not clearly-enough explained, or if you run into any problems

Good luck, and let us know how it goes!

Paul

Thanks for the reminder that Screaming Frog has that "Check Index" functionality, Thomas.

Unfortunately, I've never been able to get that method to check more than one link at a time, as all it does is send the request to a browser to check. Even highlighting multiple URLs and checking for indexation only checks the first one. Great for spot checks, but not what Seth is looking for, I don't think. My other post details an automatic way to check a site's hundreds (or thousands) of pages at a time.

I only have the free version of Screaming Frog on this machine at the moment so would be very interested to know if the paid version changes this.

Paul

There is no individual tool capable of providing the info you're looking for, Seth. At least as far as I've ever come across.

HOWEVER! It is possible to do it if you are willing to do some of the work on your own to collect and manipulate data using several tools. Essentially this method automates the approach Takeshi has mentioned.

The short answer
First you'll create a list of all the pages on your website. Then you'll create a list of all the URLs that Google says are indexed. From there, you will use Excel to subtract the indexed URLs from the known URLs, leaving a list of non-indexed URLS, which is what you asked for.

Ready? Here's how.

Collect a list of all your site's pages You can do this in several ways. If you have a reliable and complete sitemap, you can get this data there. If your CMS is capable of outputting such a list, great. If neither of these is an option, you can use the Screaming Frog spider to get the data (remember the free version will only collect up to 500 pages). Xenu Linksleuth is also an alternative. Put all these URLs into a spreadsheet.

Collect a list of all pages Google has indexed.
You'll do this using a scraper tool that will "scrape" all the URLs off a Google SERP page. There are many tools to do this; which one is best will depend largely on how big your site is. Assuming your site is only 7 or 800 pages, I recommend the brilliantly simple SERPS Redux bookmarklet from Liam Delahunty.

Clicking on the bookmarklet while on a SERP page will automatically scrape all the URLs into an easily copyable format. The trick is, you want the SERP page to display as many results as possible, otherwise you'll have to iterate through many, many pages to catch everything.

So - pro tip - if you go to the setting icon while on any Google search page, and select Search Settings you will see the option to have your searches return up to 100 results instead of the usual 10. You have to select Never Show Instant Results in order for the Results per Page slider to become active.

Now, in Google's search box, you'll enter site:mysite.com as Takeshi explained. (NOTE: use the canonical version of your domain, so include the www if that's the primary version of your site) You should now have a page listing 100 URLs of your site that are indexed.

• Click the SERPRedux bookmarklet to collect them all, then copy and paste the URLs into a spreadsheet.
• Go back to the site:mydomain results page, click for page 2, and repeat, adding the additional URLs to the same spreadsheet.
• Repeat this process until you have collected all the URLs Google lists

Remove duplicates to leave just un-indexed URLs
Now you have a spreadsheet with all known URLs and all indexed URLs. Use Excel to remove all the duplicates, and what you will be left with is all the URLs that Google doesn't list as being indexed.

Voila !

A few notes:

• The site: search operator doesn't guarantee that you'll actually get all indexed URLs, but it's the closest you'll be able to get. For an interesting experiment, re-run this process with the non-canonical version of your site address as well, to see where you might be indexed for duplicates.
• If your site is bigger, or you will need to do this multiple times, there are tools that will scrape all the SERPS pages at once so you don't have to iterate through them. The scraper components of SEER's SEO Toolbox or Neil Bosma's SEO Tools for Excel are good starting points. There is also a paid tool called ScrapeBox designed specifically for this kind of scraping. It's a blackhat tool, but in the right hands, is also powerful for whitehat purposes
• Use Takeshi's suggestion of running some of the resulting non-indexed list through manual site: searches to confirm the quality of your list

Whew! I know that's a lot to throw at you as an answer to what probably seemed like a simple question, but I wanted to work through the steps for you, rather than just hint at how it could be done.

Be sure to ask about any of the areas where my explanation isn't clear enough.

Paul

Thomas, as Takeshi has tried to point out, you have misread the original question. The original poster is asking for a way to find the actual URLS of pages from his site that are NOT indexed in the search engines.

He is not looking for the number of URLS that are indexed.

None of the tools you have repeatedly mentioned are capable of providing this information, which is likely why you're response was downvoted.

Best to carefully read the original question to ensure you are answering what is actually being asked, rather than what you assume is being asked. Otherwise you add significant confusion to the attempt to provide an answer to the original poster.

Paul