Welcome to the Q&A Forum

What an aggravating mess to have to clean up!

You're going to need to give Google stronger and less ambiguous directions for what to do with the offending domain. Right now, with the HTTPS version not redirecting properly, G is getting contradictory signals that will cause it to leave the HTTPS version in the index and prolong your issues.

The first step is to add an SSL certificate for the spam domain to the client's primary domain. I know this sounds counterintuitive, but it is required procedure in order for the redirect of the HTTPS version of the spam domain to work.

When a browser (or the Google crawler) tries to follow that HTTP redirect, the very first thing it must do is negotiate the connection. Because it's an HTTPS URL, that connection is going to fail because there is no SSL certificate present. Which means Googlebot can never get to the point where it can even see the 301 redirect. So Google thinks the spam domain is still active and leaves it in the index.

So step one is to get an SSL certificate for the www and non-www versions of the spam domain installed on the primary domain so the redirect will work.

The next step is to get the spam domain verified in its own Google Seach Console so that you can give Google direct instructions regarding what you want it to do with that domain.

Since you've deleted the spam domain's hosting, you'll need to Verify the spam domain in Google Search Console by using the DNS verification method where you add a TXT record to the spam domain's DNS records.

Once you have the spam domain verified and the HTTPS redirect is working properly, you can use the Change of Address tool in Search Console to point the spam domain to the primary domain. (The CoA tool must see a working 301 redirect in order to process the change.)

Note! At this point, I would use the Change of Address rather than the Remove URL tool in order to push any relevance signals of the spam domain to the primary instead of just dropping them altogether.

The additional benefit of the Change of Address is that the old spam domain will almost immediately drop out of the SERPs.

As a final step, it would be beneficial to get the individual spam URLs redirecting to their equivalents on the primary site. If the spam site used exactly the same URL structure, your domain redirect will take care of this. But otherwise, it would be beneficial to harvest as many of the spam URLs as possible and add 301 redirects for them into the htaccess file of the primary site. Again, we're trying to give Googlebot as many unambiguous signals as possible about what to do with the authority of those spam URLs.

Does that all make sense?

The short answer is - you can track up to 10 websites on your Medium subscription, @kassandrasharr .

That's the staging URL for GoDaddy-hosted sites, Ryan. It likely means that when the site was being developed under the hosting account's temporary URL, it got indexed by the search engines.

It's also possible there's a domain resolution issue on the hosting server. Are there any other pages showing this problem?

The reason its display is broken is because the CSS and JavaScript on the page will be getting called from the wrong URL, so they can't resolve and format the page content.

Paul

Knowing that with a large body of documentation like this, the chances of being able to rewrite it all to combine into a single page are pretty slim (and knowing that might be a very negative user experience) you're really only left with the canonical tag option - assuming the older docs need to be maintained.

You're right to be concerned, as Google has been clear that canonical only applies to pages that have substantially identical content. Unfortunately, they do a really poor job of explaining just how much variation would be allowed.

Is it okay if the canonical is not an exact duplicate of the content?
We allow slight differences, e.g., in the sort order of a table of products. We also recognize that we may crawl the canonical and the duplicate pages at different points in time, so we may occasionally see different versions of your content. All of that is okay with us.
~ https://webmasters.googleblog.com/2009/02/specify-your-canonical.html

My impression is that they would honour canonical in your use case.

Really, the only way to know is to select a couple of products' documentation pages and conduct a test. Canonicalise all old version to the current version and request re-indexing for each page. Then monitor the results (The new index monitoring tools in the new GSC are useful for this). You'll want to choose at least one test case that involves featured snippets - it would be incredibly useful to know if the FS transfers across to the new canonical page!

Do note that you'll need an ongoing process for managing the canonicals s each new iteration of documentation is added - all related pages will need to have their canonicals updated to point to the newest each time new docs are published.

Interesting conundrum. Please let us know the results if you decide to try a test!

Is that useful?

Paul

Please don't hijack someone else's question, zecase. Much better if you start your own question as it's completely unrelated to the current one.

Paul

Absolutely do NOT change from the www just for the sake of saving four characters in the URL. It would mean that every single page on your site would now have to go through a 301-redirect.

You should only change URLs when absolutely necessary, and the reasoning for this change certainly doesn't qualify!

Also worth noting - a www URL has more technical flexibility. Those reasons may not apply to you now but could in the future.

Paul

Your understanding of canonical tags is correct, GhillC.

If Tools are showing errors for those canonical tags you've listed, then the tools are wrong.

As long as the protocol and subdomain prefix (or not) exactly match and the only difference is the exclusion of the parameters (the "?" and the stuff after it) then the canonicals are correct.

Any tool's reports have to be filtered through your own understanding and knowledge. They often get things wrong. That's on eof the key differences between experienced SEOs and less-experienced. They kow when to question what an automated tool is telling them. So good on ya for questioning the results!

Paul

Direct traffic isn't a ranking factor, VT. There's no way it can be, as it would far to easy a signal to fake and abuse.

Paul

A proper canonical tag does a lot more than "just be telling Google not to rank it" When used properly (i.e. pages that truly do contain the same content), the canonicalised page passes its ranking signals back to the canonical source.

I agree with Kristina - while a 301 would be preferable (it's a directive, while canonical tags are taken as suggestions), a canonical tag would be vastly better than not doing anything about the issue. At least until the dev can get the problem with the 301-redirect properly resolved.

Paul

My suspicion would be that those URLs are accessible somewhere without the second variable. (I.e. That second variable isn't always present when the first variable is.)

There is no individual tool capable of providing the info you're looking for, Seth. At least as far as I've ever come across.

HOWEVER! It is possible to do it if you are willing to do some of the work on your own to collect and manipulate data using several tools. Essentially this method automates the approach Takeshi has mentioned.

The short answer
First you'll create a list of all the pages on your website. Then you'll create a list of all the URLs that Google says are indexed. From there, you will use Excel to subtract the indexed URLs from the known URLs, leaving a list of non-indexed URLS, which is what you asked for.

Ready? Here's how.

Collect a list of all your site's pages You can do this in several ways. If you have a reliable and complete sitemap, you can get this data there. If your CMS is capable of outputting such a list, great. If neither of these is an option, you can use the Screaming Frog spider to get the data (remember the free version will only collect up to 500 pages). Xenu Linksleuth is also an alternative. Put all these URLs into a spreadsheet.

Collect a list of all pages Google has indexed.
You'll do this using a scraper tool that will "scrape" all the URLs off a Google SERP page. There are many tools to do this; which one is best will depend largely on how big your site is. Assuming your site is only 7 or 800 pages, I recommend the brilliantly simple SERPS Redux bookmarklet from Liam Delahunty.

Clicking on the bookmarklet while on a SERP page will automatically scrape all the URLs into an easily copyable format. The trick is, you want the SERP page to display as many results as possible, otherwise you'll have to iterate through many, many pages to catch everything.

So - pro tip - if you go to the setting icon while on any Google search page, and select Search Settings you will see the option to have your searches return up to 100 results instead of the usual 10. You have to select Never Show Instant Results in order for the Results per Page slider to become active.

Now, in Google's search box, you'll enter site:mysite.com as Takeshi explained. (NOTE: use the canonical version of your domain, so include the www if that's the primary version of your site) You should now have a page listing 100 URLs of your site that are indexed.

• Click the SERPRedux bookmarklet to collect them all, then copy and paste the URLs into a spreadsheet.
• Go back to the site:mydomain results page, click for page 2, and repeat, adding the additional URLs to the same spreadsheet.
• Repeat this process until you have collected all the URLs Google lists

Remove duplicates to leave just un-indexed URLs
Now you have a spreadsheet with all known URLs and all indexed URLs. Use Excel to remove all the duplicates, and what you will be left with is all the URLs that Google doesn't list as being indexed.

Voila !

A few notes:

• The site: search operator doesn't guarantee that you'll actually get all indexed URLs, but it's the closest you'll be able to get. For an interesting experiment, re-run this process with the non-canonical version of your site address as well, to see where you might be indexed for duplicates.
• If your site is bigger, or you will need to do this multiple times, there are tools that will scrape all the SERPS pages at once so you don't have to iterate through them. The scraper components of SEER's SEO Toolbox or Neil Bosma's SEO Tools for Excel are good starting points. There is also a paid tool called ScrapeBox designed specifically for this kind of scraping. It's a blackhat tool, but in the right hands, is also powerful for whitehat purposes
• Use Takeshi's suggestion of running some of the resulting non-indexed list through manual site: searches to confirm the quality of your list

Whew! I know that's a lot to throw at you as an answer to what probably seemed like a simple question, but I wanted to work through the steps for you, rather than just hint at how it could be done.

Be sure to ask about any of the areas where my explanation isn't clear enough.

Paul

Sorry, I have to categorically disagree with Sangeetac here. Kimberley, your understanding is entirely correct.

There's so much misinformation and misunderstanding about canonicalization, and there doesn't really need to be.

The rel canonical tag is ONLY to be used when there is a justifiable reason why the multiple duplicate pages in question should actually be reachable via different URLs. Unless such a reason exists, you should ALWAYS use a 301 redirect.

Put another way, duplicate page issues should ALWAYS be resolved using 301 redirects unless doing so would harm the user experience.

In the case of multiple versions of a home page, there is absolutely NO BENEFIT to the user to be able to reach that page via multiple different URLs. (In fact the multiple URLs could be confusing.) Therefore, the 301 redirect should be used to FORCE the user (and search engine) to the single version the site owner has decided should be the primary URL. Again, the other URLs are pointless, so the user should be blocked from ever being able to reach them. Using the 301 in this case has the added benefit of explicitly and automatically telling the search engines that all the redirected URLs for that page should be dropped from the index as they are non-functional. (And that's exactly what happened, Kimberley, as you saw default.asp and index.htm disappear from the index.)

The ONLY time you'd use a rel canonical tag in this case, instead of a 301 redirect, is if for some reason the site owner doesn't have sufficient access to the sever files to be able to implement a 301 redirect properly. In that case, a rel canonical tag can be used as a "better than nothing" solution.

The classic example of when to use a rel canonical tag is in the case of a page that lists a large number of products. That default (and therefore canonical version) version of the page may very well list the products in the order they were added to inventory. There may well be additional versions of the page, each at a different URL (often using query parameters, for example) which lists exactly the same content, but one page sorted by size, another version sorted by colour, yet another sorted by price.

Obviously in this case, 301-redirecting those additional dupe pages to the original page would be harmful to the user experience, because the user would never be able to access the content sorted the way they want to see it. So each duplicate version of the page has value to the user and therefore MUST remain available to the user, and therefore to the search engine.

In this case, the site owner adds a rel canonical tag to each of the dupe pages pointing to the primary (canonical) version, to make it clear to the search engine that the dupe pages are intentional secondary variations that should pass all their ranking value back to the primary page. But NOTE! Google themselves say they consider the rel canonical tag as a suggestion only, and will ignore it if they deem it incorrect or manipulative. So it is NOT NEARLY as strong an indicator as a 301 redirect which forces both user and search engine into the desired behaviour.

Sorry if that got a little long-winded, but once we get clear on the basic purpose of 301 redirects vs rel canonical tags, it becomes much more straightforward which to use when. The confusion usually stems from folks trying to justify using canonical tags for purposes for which they were never designed. They are not a panacea or a catch-all for fixing site architecture mistakes.

Hope that clears, rather than muddies the waters?

Paul

I'm going to have to offer a different opinion than Chad, njam. There's nothing about tags that is intrinsically beneficial for SEO, and in fact misusing them can cause considerable SEO problems.

Bottom line, the primary purpose of tags is as an additional navigation tool for your visitors, and must be carefully implemented so not to damage your site's SEO. There's nothing magic about a tag - it's a link to your content organized in different ways

Tags are part of what's called your site's taxonomy - i.e. how it organizes its information. Tags allow the visitor to get listings of your existing content grouped by different topics - much like categories do, but on a more granular scale. And there's the kicker - tag listings are just your regular content, but grouped differently.

Because they're the same content as exists elsewhere on the site, they can easily be seen as duplicate content by the search engines, leading to a dilution of the ranking power of the actual posts. (This issue can also occur with category listings if they're not handled correctly, which is why you're already no-indexing them).

Most WordPress SEO specialists actually recommend that tag pages be no-indexed - a command that tells the search engines to not even include them in search results as they can so easily confuse the search engines about which version of your site's content is most important.

So... use tags on a website if you think they will significantly add to the visitors' ability to understand and navigate your site. Then, make sure you've taken steps to ensure they don't confuse the search engines.

Do NOT use tags just because they're available. If you don't have a clear strategy for using them, don't clutter your site with them.

If you want an idea of how tags fit into a taxonomy, here's a real-work analogy:

Think of your blog as a book on raising kids, for example. The book would have a bunch of separate chapters on things like childbirth, schooling and education, care and feeding, discipline, family finances - you get the drift.

These are the equivalent to your blog's categories - broad topics that would allow someone to pick up your book and leaf through it, immediately understanding what the book's about, and enabling them to read just the sections that apply to them.

In the back of the book is the index. It's full of words and phrases that you have used a couple times in the book, but aren't topics that deserve a whole chapter (category). These are the equivalent of your tags. For example, perhaps you mention peanut butter in the care & feeding chapter. Later, you also mention in the schooling chapter that parents shouldn't send peanut butter sandwiches in kids' lunches because of the allergy risk to others. In addition, you mention PB in passing in the finance chapter as a cheap source of protein for kids meals.

Obviously there'd be no point in having a whole chapter (category) on peanut butter, but a reader might find it useful to have the index (tags) at the back of the book show them all the places in your book you've mentioned peanut butter.

That's a bit of a facetious example, but hopefully it gives you an idea of the different uses? If not, lemme know and I'll try to clarify further. Just remember, there's nothing magic about tags for SEO and they in fact carry some risk. They are for human visitors. If the visitors don't find them useful, they shouldn't be there.

Paul

It's not unusual at all to experience this kind of minor, temporary drop, Gordian. The search engines will need to re-index the changes in content and URLs created by your redesign, and this can easily take a week or two.

The critical thing is to be certain you have effective monitoring processes in place to make certain you catch any issues that the redesign might have created just a soon as they occur, so you can fix them before any long-term damage is done.

• Segment out your traffic to determine exactly what source or sources might have caused the drop. For example, your overall traffic may have dropped, but if you look at the source segments, it may turn out that your organic search traffic is consistent, but your site's referral traffic from Pinterest has dropped significantly. (I've had this exact situation with a client - a goof in the .htaccess file was breaking the Pinterest referrals)
• Use Google Analytics and Webmaster Tools to make certain you're catching any new 404 errors and getting them fixed ASAP
• Use Google Analytics to monitor your most valuable pages to make sure they're not seeing an unusual increase in bounce rate, or unexpected drop in time on page.
• Use Google and Bing Webmaster Tools to ensure your site is being fully crawled and watch for changes to how many pages are indexed. (You can use the "submit URL" tools in both to submit the most important pages of the site for re-crawl - can often speed up the process of getting the new URL structure recognized & indexed.)
• Watch for unhealthy increases in page load times for your most important pages via Google Analytics. (Note: you have to customize the Analytics code snippet to get it to include more than the default 10% of pageviews which is almost never enough for accurate analysis.)
• Use your Moz Analytics crawls to keep an eye out for unexpected ranking drops in your most important keywords. (GWT average rankings can also be used for this) Also watch for any increase in dupe content flagged in Moz Analytics

The idea is that yes - it's natural to see a small, temporary drop after a redesign. But you want to be certain that the drop isn't being caused by a correctable technical issue. Hence the need for close monitoring, even if assuming it's just temporarily due to new site design.

Hope that all makes sense?

Paul

P.S. Getting some new, quality, authoritative links to the newly designed pages can really help too. Social Media, especially Google+ can be really effective for this.

There is only one way the use of Hootsuite could negatively affect SEO - if you choose the ow.li URL shortener as your default instead of the default ow.ly version (in the setup section).

The ow.li version is using 302 redirects instead of 301s which is a no-no for SEO.

Otherwise, since the dashboard is only posting contents to Twitter/Facebook and not your website, there's no possibility for harm.

HOWEVER!!! Using Hootsuite (or any 3rd-party tool) to post to Facebook can definitely cause user-engagement problems. Any update posted to Facebook using a 3rd-party tool (like a dashboard) automatically has a harder time earning views by your followers. Facebook's Edgerank algorithm is what determines how many of your followers might see your post, and posts by 3rd party tools always have a lower Edgerank.

So for all important Facebook posts, it is best to submit them using Facebook's own site or their mobile app rather than Hootsuite. (You can now pre-schedule posts from within Facebook itself, so this helps alleviate one of the reasons why most people used 3rd-party tools) In fact, I still believe it's better to make all Facebook updates directly.

Paul

Thomas, as Takeshi has tried to point out, you have misread the original question. The original poster is asking for a way to find the actual URLS of pages from his site that are NOT indexed in the search engines.

He is not looking for the number of URLS that are indexed.

None of the tools you have repeatedly mentioned are capable of providing this information, which is likely why you're response was downvoted.

Best to carefully read the original question to ensure you are answering what is actually being asked, rather than what you assume is being asked. Otherwise you add significant confusion to the attempt to provide an answer to the original poster.

Paul

You could always just check the source of the page and count the hrefs to know how many links you actually have, Erin

Easier way would be to use the SEOMoz extension for Chrome or Firefox. It has a button for Analyze Page (looks like a magnifying glass) which has a tab for Page Attributes. That will show you the total count of internal and external links on a page. (It just does the manual counting for you, essentially)

You could also use a standalone page like http://linkcounter.submitexpress.com/ but the SEOMoz plugin/extension is so useful for many things that it's worth installing anyway.

Sounds to me like your first crawl didn't get through the pages thoroughly, but you're right to be careful. Especially in view of possible spam links, as you mentioned.

Hope that helps?

Paul

You went one step too far by blocking the login pages in the robots.txt file, Tiffany.

You've put the no-index tag in the header of each login page, which is the right way to tell Google to remove the page from its index.

But by also blocking the subdomains in robots.txt, you've told the search engines not to spider the pages, so there's no way they can discover the no-index on the individual pages. They can't obey commands on pages they're told not to look at.

To be clear - a block in robots.txt does NOT tell the SEs to remove the pages, it just says "don't crawl them". You need to allow the pages to be crawled so the SEs can find and obey the no-index directive to remove them.

Doing that will just be a slower, though obviously much more automatic, way of accomplishing the same result as requesting a manual removal for each the URLs. Once you can see that all the subject URLs have fallen out of the index, you can reapply the robots.txt block to help save crawl budget, but keep the no-index meta-tags on each page as extra insurance.

Does that makes sense?

Paul

You're in a delicate spot, Bob. I'd say your plan should be to "hope for the best, but plan for the worst".

Obviously, as you indicated, you're going to need to do something about those links as that link profile is just begging to get hammered.

You could clean them all at once, take the traffic hit, and then try to build back as quick as possible, but if the site is doing well now, it seems a shame to take such a hit.

I'd suggest putting a clear, well-prioritized, well-funded plan in place to start building link-worthy content and promoting it in ways that earn those backlinks as quickly as possible. (This work is going to have to be done regardless, so not like it's a temporary expense).

Then, for every 6 or 8 new quality incoming links, clean up one of the 10 problematic links. This will look natural to the SEs (as it is natural) and hopefully won't attract the attention of the slappers while you're working through the process.

Best case scenario, you'll get through offsetting all the problem links without getting hurt by a penalty or algo update.

In order to be ready in case of the worst-case scenario, (Google slaps the site with a penalty a week from now), you should also immediately build a confirmed contact list of the webmasters in control of the problematic links. (I mean an email or phone number that you've confirmed actually gets a response from a human). That way if you get hit before you can clean up naturally, you can get those problem links dealt with immediately and can show Google what you've done in a quick reconsideration request.

Also, document the process as you work through attracting the new links, so you can be specific about what you've been doing in that direction, should a reconsideration request become necessary

As far as how to deal with the problem links - do not submit a disavow!! That is a last-ditch process if there's no other way to get links removed, which is not your case. (Plus the disavow process could attract unwanted attention. Yea, I'm cynical like that

I'd actually suggest a mix of tactics for those 10 sites, depending on different circumstances:

• If a site's links are generating quality traffic, just ask that they be no-followed.
• If using the no-follow approach on a number of the sites, also see if they can mix up the anchor text, making sure to include at least some branded (as you hinted)
• If the main value of the links is for juice, and the site is relevant to your own, ask that they be swapped for a legit editorial link or two. A couple of the strong, new, link-worthy content resources you've just built will help here. (And will probably be stronger than a sitewide anyway)
• If the links aren't generating quality traffic and aren't relevant to your niche, just get them removed.

Does that approach sound like it might work?

Paul

You're right, GA data can be overwhelming if you've never looked at it before, Icarus. I almost always find the most useful place to start is the Channels report **Acquisition > All Traffic > Channels. **It will show you your current traffic divided out by the primary sources, and will also show you the comparable bounce rate, time on page etc for each.

The other report that's pretty well always essential is the Content report Behaviour > Site Content > All Pages. This will show you which individual pages are getting the most visits on the site.

Use the date selector in the top right corner to review the last month's data, and then to compare it to the previous month, or the same period of the previous year. The latter is important for sites that have seasonally-variable traffic, as it might not be sensible to compare Christmas traffic to November, for example.

Beyond that, it's going to depend on why type of site (e-commerce, lead gen, publishing) and what elements of your site you're working on to improve, to determine what might be most useful to look at.

And for the bonus, no, nothing in Analytics will tell you about your competitors. GA is for showing you what's happening on your own site.

Hope that gets you started?

Paul

P.S. Once you get going with your data, you're going to want to set up some shortcuts (quick ways to navigate to reports that usually take a number of clicks,) and Dashboards (collections of information you've decided you want to check frequently in one place instead of having to navigate through the interface each time.)