Your understanding of canonical tags is correct, GhillC.

If Tools are showing errors for those canonical tags you've listed, then the tools are wrong.

As long as the protocol and subdomain prefix (or not) exactly match and the only difference is the exclusion of the parameters (the "?" and the stuff after it) then the canonicals are correct.

Any tool's reports have to be filtered through your own understanding and knowledge. They often get things wrong. That's on eof the key differences between experienced SEOs and less-experienced. They kow when to question what an automated tool is telling them. So good on ya for questioning the results!

Paul

A proper canonical tag does a lot more than "just be telling Google not to rank it" When used properly (i.e. pages that truly do contain the same content), the canonicalised page passes its ranking signals back to the canonical source.

I agree with Kristina - while a 301 would be preferable (it's a directive, while canonical tags are taken as suggestions), a canonical tag would be vastly better than not doing anything about the issue. At least until the dev can get the problem with the 301-redirect properly resolved.

Paul

I'd absolutely agree. If you can get it fixed properly in a couple of months, then leaving the status quo would e better than making interim changes that just trade one set of redirects for another.

P.

Honestly? I'd spend the time to get the Custom CMS fixed to allow trailing slashes in the navigation links. That would eliminate the redirect issue, Instead of just trading it off to another set of links that would have to redirect.

It sounds like a code sanitising issue in the CMS. Worth spending a couple of hundred dollars to fix the root cause of the issue instead of spending that money to apply bandaids that cause other problems elsewhere. (And bonus, maybe you can get proper canonicalisation built at the same time.)

Of course, yea, this does depend on having/finding a competent developer and having a test environment that doesn't endanger the live site.

Any chance you could push for this option?

Paul

Screaming Frog also has a basic, useful site visualisation capability built into it.

So much this ^^

This is a question for which the only answer is "test your own visitors".

Something else to keep in mind - so much of getting visitors to complete longer forms is about setting expectations and demonstrating WIIFM (what's in it for me) up front.

"A minute and half of your time for these questions helps us give you more useful customised recommendations. We don't want to waste your time." Can go an awful long way to getting buy-in from visitors to complete the form. You've told them why giving the info will help them, and told them how much time is required. So even if they see a multi-step form, they won't get worried it's going to drag on.

Something else to think about.

Paul

One more sidenote - you can also use inbound link auditing tools like Moz Open Site Explorer, AHREFS, or especially Cognitive SEO to collect as many of your incoming links as possible, then filter them for which ones 404 and then you'll know which external pages contain the broken incoming links. CognitiveSEO is especially well set up to do this, but it's not a cheap tool (Does have a free trial though).

Hope that helps?

Paul

While essential to maintaining the quality of the website, remember this process will only show missing URLs as detected from within the website. It can't show any info about all the broken links that may exist externally on other websites.

P.

Agree with Stephan - the dupe H1s, while not ideal, are not going to be a significant SEO issue.

But what will be the issue is the extra page bloat and therefore load speed of all those pages. Not necessarily for SEO, but most certainly for usability. There are much more efficient ways to get your pages to scale for different devices than to run three sets of headers.

Paul

These tools are designed to report very different things, Eric. SF and Moz are only able to tell you what they can discover about what's happening within your site, since that's all they have access to. (Moz's link index isn't tied into their site crawl reports). So your own pages that are linking to other pages which no longer exist.

Search Console is also showing what's happening on the rest of the web outside your website. So it's picking up and reporting on links on other websites that it's trying to follow but finding 404 at the end on your site.

So, for example, Search Console is the only one that's going to be able to find and report on URLs that you used to have, but that haven't been correctly redirected after the URLs changed, like in a redesign. Assuming you cleaned up your internal linking after the redesign, the only way to know that these other URLs still haven't been properly redirected is via GSC's external visibility.

If GSC is finding a lot of these, and you want to proactively find as many more as possible without waiting for GSC to find them. there are a couple of methods available to try to address in bulk. One way is to run a Screaming Frog crawl of the previous version(s) of the site in the Wayback Machine, then rerun those as a list to discover which ones now 404. Or if you have an old sitemap, you can upload that into SF and run that and check for 404s.

A partial alternative is to run a custom report in GA that shows the referrer to the 404 page. But it will only track actual clickthroughs, whereas GSC actually reports on external broken links, whether they've been clicked on recently or not. So much broader and a bit more proactive. Eoghan at Rebelytics has a great post on assembling these useful reports, including links to directly import from the Solutions Gallery. Do note though, this data can often be pretty "dirty" with direct/(not set) referrers, or very vague ones like "Facebook"

But bottom line - use your Screaming Frog crawl to find and fix the broken links within your own site. USe the GSC list (downloaded) to write appropriate redirects for the URLs it finds that haven't' been covered by your internal fixes. Since you can't do anything about the broken links on third-party sites, the only control you have of them is writing redirects inside your own system. Put another way - it's not worth the effort to find where on the external webs the broken inbound links are from, since you can't do anything about changing them anyway.

Hope that all makes sense?

Paul

P.S. Marking the 404s as fixed in GSC will suppress them for about 90 days, but if the broken links still exist out on the web or in the index and are re-crawled, they will again start to show up in your GSC list until they've been crawled often enough that G decides to trust the 404 and may stop crawling them. Once you apply redirects, G will stop reporting them as 404s. Do note though that if the equivalent content really is gone, there's nothing wrong with letting the broken links 404. Trying to artificially redirect them to unrelated pages (like the home page) will just result in G treating them as (soft)404s anyway.

There is no individual tool capable of providing the info you're looking for, Seth. At least as far as I've ever come across.

HOWEVER! It is possible to do it if you are willing to do some of the work on your own to collect and manipulate data using several tools. Essentially this method automates the approach Takeshi has mentioned.

The short answer
First you'll create a list of all the pages on your website. Then you'll create a list of all the URLs that Google says are indexed. From there, you will use Excel to subtract the indexed URLs from the known URLs, leaving a list of non-indexed URLS, which is what you asked for.

Ready? Here's how.

Collect a list of all your site's pages You can do this in several ways. If you have a reliable and complete sitemap, you can get this data there. If your CMS is capable of outputting such a list, great. If neither of these is an option, you can use the Screaming Frog spider to get the data (remember the free version will only collect up to 500 pages). Xenu Linksleuth is also an alternative. Put all these URLs into a spreadsheet.

Collect a list of all pages Google has indexed.
You'll do this using a scraper tool that will "scrape" all the URLs off a Google SERP page. There are many tools to do this; which one is best will depend largely on how big your site is. Assuming your site is only 7 or 800 pages, I recommend the brilliantly simple SERPS Redux bookmarklet from Liam Delahunty.

Clicking on the bookmarklet while on a SERP page will automatically scrape all the URLs into an easily copyable format. The trick is, you want the SERP page to display as many results as possible, otherwise you'll have to iterate through many, many pages to catch everything.

So - pro tip - if you go to the setting icon while on any Google search page, and select Search Settings you will see the option to have your searches return up to 100 results instead of the usual 10. You have to select Never Show Instant Results in order for the Results per Page slider to become active.

Now, in Google's search box, you'll enter site:mysite.com as Takeshi explained. (NOTE: use the canonical version of your domain, so include the www if that's the primary version of your site) You should now have a page listing 100 URLs of your site that are indexed.

• Click the SERPRedux bookmarklet to collect them all, then copy and paste the URLs into a spreadsheet.
• Go back to the site:mydomain results page, click for page 2, and repeat, adding the additional URLs to the same spreadsheet.
• Repeat this process until you have collected all the URLs Google lists

Remove duplicates to leave just un-indexed URLs
Now you have a spreadsheet with all known URLs and all indexed URLs. Use Excel to remove all the duplicates, and what you will be left with is all the URLs that Google doesn't list as being indexed.

Voila !

A few notes:

• The site: search operator doesn't guarantee that you'll actually get all indexed URLs, but it's the closest you'll be able to get. For an interesting experiment, re-run this process with the non-canonical version of your site address as well, to see where you might be indexed for duplicates.
• If your site is bigger, or you will need to do this multiple times, there are tools that will scrape all the SERPS pages at once so you don't have to iterate through them. The scraper components of SEER's SEO Toolbox or Neil Bosma's SEO Tools for Excel are good starting points. There is also a paid tool called ScrapeBox designed specifically for this kind of scraping. It's a blackhat tool, but in the right hands, is also powerful for whitehat purposes
• Use Takeshi's suggestion of running some of the resulting non-indexed list through manual site: searches to confirm the quality of your list

Whew! I know that's a lot to throw at you as an answer to what probably seemed like a simple question, but I wanted to work through the steps for you, rather than just hint at how it could be done.

Be sure to ask about any of the areas where my explanation isn't clear enough.

Paul

Sorry, I have to categorically disagree with Sangeetac here. Kimberley, your understanding is entirely correct.

There's so much misinformation and misunderstanding about canonicalization, and there doesn't really need to be.

The rel canonical tag is ONLY to be used when there is a justifiable reason why the multiple duplicate pages in question should actually be reachable via different URLs. Unless such a reason exists, you should ALWAYS use a 301 redirect.

Put another way, duplicate page issues should ALWAYS be resolved using 301 redirects unless doing so would harm the user experience.

In the case of multiple versions of a home page, there is absolutely NO BENEFIT to the user to be able to reach that page via multiple different URLs. (In fact the multiple URLs could be confusing.) Therefore, the 301 redirect should be used to FORCE the user (and search engine) to the single version the site owner has decided should be the primary URL. Again, the other URLs are pointless, so the user should be blocked from ever being able to reach them. Using the 301 in this case has the added benefit of explicitly and automatically telling the search engines that all the redirected URLs for that page should be dropped from the index as they are non-functional. (And that's exactly what happened, Kimberley, as you saw default.asp and index.htm disappear from the index.)

The ONLY time you'd use a rel canonical tag in this case, instead of a 301 redirect, is if for some reason the site owner doesn't have sufficient access to the sever files to be able to implement a 301 redirect properly. In that case, a rel canonical tag can be used as a "better than nothing" solution.

The classic example of when to use a rel canonical tag is in the case of a page that lists a large number of products. That default (and therefore canonical version) version of the page may very well list the products in the order they were added to inventory. There may well be additional versions of the page, each at a different URL (often using query parameters, for example) which lists exactly the same content, but one page sorted by size, another version sorted by colour, yet another sorted by price.

Obviously in this case, 301-redirecting those additional dupe pages to the original page would be harmful to the user experience, because the user would never be able to access the content sorted the way they want to see it. So each duplicate version of the page has value to the user and therefore MUST remain available to the user, and therefore to the search engine.

In this case, the site owner adds a rel canonical tag to each of the dupe pages pointing to the primary (canonical) version, to make it clear to the search engine that the dupe pages are intentional secondary variations that should pass all their ranking value back to the primary page. But NOTE! Google themselves say they consider the rel canonical tag as a suggestion only, and will ignore it if they deem it incorrect or manipulative. So it is NOT NEARLY as strong an indicator as a 301 redirect which forces both user and search engine into the desired behaviour.

Sorry if that got a little long-winded, but once we get clear on the basic purpose of 301 redirects vs rel canonical tags, it becomes much more straightforward which to use when. The confusion usually stems from folks trying to justify using canonical tags for purposes for which they were never designed. They are not a panacea or a catch-all for fixing site architecture mistakes.

Hope that clears, rather than muddies the waters?

Paul

It's not unusual at all to experience this kind of minor, temporary drop, Gordian. The search engines will need to re-index the changes in content and URLs created by your redesign, and this can easily take a week or two.

The critical thing is to be certain you have effective monitoring processes in place to make certain you catch any issues that the redesign might have created just a soon as they occur, so you can fix them before any long-term damage is done.

• Segment out your traffic to determine exactly what source or sources might have caused the drop. For example, your overall traffic may have dropped, but if you look at the source segments, it may turn out that your organic search traffic is consistent, but your site's referral traffic from Pinterest has dropped significantly. (I've had this exact situation with a client - a goof in the .htaccess file was breaking the Pinterest referrals)
• Use Google Analytics and Webmaster Tools to make certain you're catching any new 404 errors and getting them fixed ASAP
• Use Google Analytics to monitor your most valuable pages to make sure they're not seeing an unusual increase in bounce rate, or unexpected drop in time on page.
• Use Google and Bing Webmaster Tools to ensure your site is being fully crawled and watch for changes to how many pages are indexed. (You can use the "submit URL" tools in both to submit the most important pages of the site for re-crawl - can often speed up the process of getting the new URL structure recognized & indexed.)
• Watch for unhealthy increases in page load times for your most important pages via Google Analytics. (Note: you have to customize the Analytics code snippet to get it to include more than the default 10% of pageviews which is almost never enough for accurate analysis.)
• Use your Moz Analytics crawls to keep an eye out for unexpected ranking drops in your most important keywords. (GWT average rankings can also be used for this) Also watch for any increase in dupe content flagged in Moz Analytics

The idea is that yes - it's natural to see a small, temporary drop after a redesign. But you want to be certain that the drop isn't being caused by a correctable technical issue. Hence the need for close monitoring, even if assuming it's just temporarily due to new site design.

Hope that all makes sense?

Paul

P.S. Getting some new, quality, authoritative links to the newly designed pages can really help too. Social Media, especially Google+ can be really effective for this.

There is only one way the use of Hootsuite could negatively affect SEO - if you choose the ow.li URL shortener as your default instead of the default ow.ly version (in the setup section).

The ow.li version is using 302 redirects instead of 301s which is a no-no for SEO.

Otherwise, since the dashboard is only posting contents to Twitter/Facebook and not your website, there's no possibility for harm.

HOWEVER!!! Using Hootsuite (or any 3rd-party tool) to post to Facebook can definitely cause user-engagement problems. Any update posted to Facebook using a 3rd-party tool (like a dashboard) automatically has a harder time earning views by your followers. Facebook's Edgerank algorithm is what determines how many of your followers might see your post, and posts by 3rd party tools always have a lower Edgerank.

So for all important Facebook posts, it is best to submit them using Facebook's own site or their mobile app rather than Hootsuite. (You can now pre-schedule posts from within Facebook itself, so this helps alleviate one of the reasons why most people used 3rd-party tools) In fact, I still believe it's better to make all Facebook updates directly.

Paul

Thomas, as Takeshi has tried to point out, you have misread the original question. The original poster is asking for a way to find the actual URLS of pages from his site that are NOT indexed in the search engines.

He is not looking for the number of URLS that are indexed.

None of the tools you have repeatedly mentioned are capable of providing this information, which is likely why you're response was downvoted.

Best to carefully read the original question to ensure you are answering what is actually being asked, rather than what you assume is being asked. Otherwise you add significant confusion to the attempt to provide an answer to the original poster.

Paul

You went one step too far by blocking the login pages in the robots.txt file, Tiffany.

You've put the no-index tag in the header of each login page, which is the right way to tell Google to remove the page from its index.

But by also blocking the subdomains in robots.txt, you've told the search engines not to spider the pages, so there's no way they can discover the no-index on the individual pages. They can't obey commands on pages they're told not to look at.

To be clear - a block in robots.txt does NOT tell the SEs to remove the pages, it just says "don't crawl them". You need to allow the pages to be crawled so the SEs can find and obey the no-index directive to remove them.

Doing that will just be a slower, though obviously much more automatic, way of accomplishing the same result as requesting a manual removal for each the URLs. Once you can see that all the subject URLs have fallen out of the index, you can reapply the robots.txt block to help save crawl budget, but keep the no-index meta-tags on each page as extra insurance.

Does that makes sense?

Paul

The big benefit to having the redirects written into htaccess is that htaccess will run them much more quickly (and with lower server overhead) than from inside WordPress. If you only have a few redirects at a time to correct for moved pages or creating short URLs for marketing campaigns, doing it within the Redirection plugin is fine.

But if you're writing a large number of redirects (to handle a site move for example) you're far better off writing them into htaccess.

Paul

What Egol has just described is why SEO is only part science and the rest - a significant portion - is still an art. It's also why automated implementation tools have never been successful.

It takes juggling a wide range of constantly changing factors, many with very subjective values, to deliver success. Or to determine whether the costs of attempting success are just too high to be worth it.

Website owners always HATE to hear the accurate response, but the honest response to so many (most?) SEO questions is:

It Depends.

And the good, ethical SEOs will tell you that right up front, as Egol has done. It would be so much easier if SEO, Conversion Rate Optimization etc were linear pursuits - do X and the result will be Y. But they're not, which is why good SEOs will beat those who simply follow (and promise results from ) formulas.

Frustrating huh?

Paul

The main reason PageSpeed and YSlow recommend including width and height for images is as much the perception of page speed as the actual load time, Ben.

When you include the image dimensions, the browser can draw out the "containers" that will hold the images, reserving the space for them while they download. The browser can then go on the paint the rest of the pages CSS and objects around those "containers" without having to go back and redraw the whole page once the images have downloaded and their sizes are then known.

This gives the user the illusion of a much faster, cleaner page load, and hence the impression of a faster website.

Does that make sense?

Paul

[Edited to add: You should still keep doing what you're doing to produce "size-as" images for your pages. You don't want to be resizing images with the html dimensions, just reporting in html the actual size of the image]

_If I have some inbound links pointing to www.MY-CUSTOMER-SITE.com and some pointing to MY-CUSTOMER-SITE.com, I would think that this rewrite isn't necessary as it would seem that Googlebot is smart enough to know that these aren't two sites. _

Absolutely NOT, unfortunately. Search engines specifically consider these two versions of the URLS to be two totally different sites. The redirect rule you currently have is specifically in place to correct this problem so the two versions of your site (in the eyes of the engines) aren't competing with each other.

The previous developer knew what he was doing. Leave the redirect as-is. Just be careful that all links you create use the primary version of the URL - you'll retain a bit more "link juice" that way than having them go through the redirect. (i.e. always write links as www.my-customer-site.com/whatever for links in content, menus, incoming links where possible)

Paul

P.S. For proof that search engines consider those URLs different sites, Google's own Webmaster Tools has a setting where you can tell Google which version of the site URL you want to be primary. Much better to do this with a proper 301-redirect though so that you can tell ALL search engines, not just Google.