@jerrico1 Only including some pages in the sitemap won't hurt your SEO performance at all. I've done this on a number of sites for exactly the same reasons you are facing.

The XML sitemap simply gives Google one more way to find your pages. Ideally, you could use it to give Google a way to find all of your pages but you want to at least use it for the pages you want to be sure Google finds. However, there is no penalty if the page isn't in the sitemap.

That said - you may want to check if you need the XML sitemap at all as a point of discovery. If you have lots of links (internal or external) to the pages on your website, then odds are good that Google is already finding those pages. The XML sitemap wouldn't hurt to have but if there already links to these pages, you likely don't have a big problem to solve here.

The best way to check this is within your log file - pull a unique list of all the URLs that Google has crawled over the last few weeks. You may not be able to open up your log files (sometimes you can't easily on large sites and you aren't using an enterprise log analyzer). If that is the case, then you could check to see how many of your pages are Google organic landing pages in your analytics tool--if the page is getting traffic from Google, then Google clearly found the page.

Hope that helps!

Hi. I have a client who is about to deploy ads on their site. To avoid bots clicking on those ads and skewing data, the company would like to prevent any bots from seeing any ads and, of course, that includes Googlebot. This seems like it could be cloaking and I'd rather not have a different version of the sites for bots. However, knowing that this will likely happen, I'm wondering how big of a problem it could be if they do this. This change isn't done to manipulate Googlebot's understanding of the page (ads don't affect rankings, etc.) and it will only be a very minimal impact on the page overall.

So, if they go down this road and hide ads from bots, I'm trying to determine how big of a risk this could be. I found some old articles discussing this with some suggesting it was a problem and others saying it might be okay in some cases (links below). But I couldn't find any recent articles about this. Wondering if anybody has seen anything new or has a new perspective to share on this issue? Is it a problem if all bots (including Googlebot) are unable to see ads?

https://moz.com/blog/white-hat-cloaking-it-exists-its-permitted-its-useful
https://www.webmasterworld.com/google/4535445.htm
https://www.youtube.com/watch?v=wBO-1ETf_dY

Hi Judd. You wouldn't need the title or description tags for Google's sake (or Bing's sake) on pages behind a login since they couldn't crawl those pages. That said, title tags aren't just used by search engines. That is also the title that shows for the page in the browser window and in the bookmark name (if people bookmark those pages), so it can be worthwhile to add a good title tag to help your visitors. Obviously, beyond the meta description tag, there are other meta tags that you might need (like meta viewport or metra reresh, etc.).

Hi. If you are getting duplicate content warnings between the Mandarin version of your website and a website in a different language, the first thing I'd suggest looking into is to see if your code correctly uses the alternate/hreflang attributes to specify the different versions of the pages in different languages. This tag is intended for sites that have translated content from one language to another and gives you a way to tell Google that the Mandarin version of the page is at this URL, the Spanish version is this URL, the English site is at this other URL, and so on.

In terms of title, descriptions, and all other content, you do want to make that specific to the language of the page. So, your Mandarin website would have titles, descriptions, etc. written in Mandarin and your English website would have titles, descriptions, etc. in English, and so on. After all, your Mandarin website is intended for people speaking (and searching in) Mandarin so using that language throughout will increase your chances of ranking for that audience.

I'd argue there is value in looking at and benchmarking both numbers, though you might not get an accurate picture of both through Google Analytics. You want to know how many sessions ended up in an order, regardless of how many repeat customers there were that converted. As you said, every visit could end up in an order (you could get a little more detailed and segment to clarify just how many sessions qualify) and you want to know just how true that is. At the same time, you want to know how many unique people placed an order as well and repeat order rates.

Here is the tricky part. Google Analytics is pretty good at telling you how many sessions resulted in an order (the conversion rate you see in goal reports is goals per session). With the Time to Purchase report, you can get a fairly decent idea of sessions it took for those higher margin products. Now the other side: unique users. Users is wonky in how it is calculated (for instance one customer uses different browsers/devices or your customer deleted their cookies) so knowing how many users converted won't always give you the number you are after and, in my mind, it isn't reliable enough to benchmark.

What I do to get at the number of unique customers and orders per customer is use other tools (CRM, order system, etc.) to track that number--those systems are designed around people not sessions, so you are going to get a far more accurate picture of how many unique people placed an order. That is your benchmark, but you can map order dates and/or transaction IDs to GA so that you can understand traffic patterns for repeat customers and how they might differ.

Hope that helps.

Generally speaking, this isn't the worst problem you can have with your XML sitemap. In an ideal world, you'll be able to remove duplicate URLs from the sitemap and only submit a single URL for each page. In reality, most larger sites I've encountered have some amount of duplicate content in their XML sitemap with no real major problems.

Duplicate content is really only a major problem if it is "deceptive" in nature. So long as this is just a normal consequence of your CMS, or similar, vs. an attempt to game the rankings you are probably fine. For more about that check out this support article.

The other problem you may encounter is with your search results for those duplicate pages. That article makes mention that Google will pick the URL they think is best (more about that here as well) and the URL they deem the best will be the URL that surfaces in the search results. That may or may not be the same URL you or your visitors would deem best. So, what you might find is Google picked a not great URL (like one with extra parameters) and with the not great URL appearing in the SERPs, your search result isn't as compelling to click on as some other version of the URL may be.

Really great question and there really isn't any one right answer here. In general, from an SEO perspective, I lean toward one consolidated website that is really well organized. That way all of your SEO efforts benefit one domain. For example, instead of building links to three sites, you can concentrate on building links to one website. Even if the content for patients gets more links (let's say), if everything is on one domain those patient oriented links will still help the content for therapists earn rankings because they'll contribute to overall domain strength.

Along with the SEO though, my other question would be if there is any great harm in having one website serve multiple audiences? There are numerous examples of companies who are able to do exactly that with their content. Doing so requires a strong information architecture to clearly define what each section is, who it is for, how sections are labeled, how you navigate to various sections, etc. Totally doable, and good IA tends to also be good for SEO too. That said, in some cases one audience group might be distracted/offended/annoyed by content that is intended for another audience group or maybe there is just one set of content you'd rather one group not see. Do you have any situations like that? Have you surveyed users for their opinions about the content to identify these pitfalls?

Of course, the other question to ask here is if there is a strong business case for dividing the sites apart? It doesn't sound like it based on your question, but I want to throw that idea out there. I've worked with some organizations where they have one department focused on a certain audience group. To simplify dev and maintenance, the business case is pretty compelling to split the sites apart. Still though, in a lot of cases it is easier to have one website because then all dev, design, branding, etc. budgets (of time and money) can be focused on the one domain vs. divided across multiple domains.

I've not seen any instances of a limit to how many redirects you can have pointing to your website. I have some clients who have thousands of redirects in place (lots of old pages being moved to a new version of that product). Those sites haven't had any issues with rankings at all. In fact, many of the links pointing to the sites still reference the URLs that are redirected and those pages that are redirected to are ranking perfectly fine.

The biggest limit I've seen is on chaining. I've seen issues where chained redirects simply aren't followed. However, if you can keep it to a 1 step redirect, or 2, then things should be okay. It doesn't sound like that is what you are asking about though. More from Matt Cutts on this:
http://www.searchenginejournal.com/matt-cutts-discusses-301-permanent-redirects-limits-on-websites/46611/

In terms of managing those redirects, you can't usually keep this many on an htaccess file without going a little bit nuts (or risking some future dev deleting those in an effort to clean up the htaccess file - ug). If you are using WordPress, the 301 redirects plugin works quite well: https://wordpress.org/plugins/301-redirects/

Unfortunately, I've also run into sites that aren't in a CMS where you can use a plugin. In those cases, I usually put these redirects in a database table. On the 404 file, I then have the code check the would-be error URL to see if we need to redirect that URL somewhere else. If a redirect is place, it redirects instead of throwing the 404 error. If no redirect is in place, the code then throws a 404 error.

Hope that helps.

I think the JavaScript implementation might still be able to be crawled by Google. Any more, I'm becoming convinced that Google can crawl just about anything. But, I'll be curious to see what the results are. Definitely update this thread with what ends up happening from that approach.

As for the robots.txt message, that would indicate that they are finding the link to the page but not crawling the page to get any content.

As for duplicated content concerns, just to take a step back, are the pages 100% the same or are you making alterations to the text? If you can do easy things that make that page different from the other sites (even if it is functionality), then the page isn't a true duplicate and there might be some good reasons why people could want to find those pages in the search results.

Ultimately, you have the same page, but you are making the page better than those other websites. If that is the case, then you should be safe letting those pages rank. Where having the same content as your competitor really hurts (in my experience, anyway) is when you aren't offering anything different than any other sites.

Hope that helps.

Hey,

This is definitely a complicated issue, and there is some risk in making a move in the wrong direction.

Here are my thoughts which might help you out. Feel free to private message me or shoot me an email (see my profile) and I'd be happy to talk more.

On the hash solution, would that require JavaScript be enabled in order to access those pages or would you have a fallback solution for those without JavaScript?

If you don't have a fallback solution for those without JavaScript, you might negatively affect visitors with disabilities. For instance, some types of Ajax are challenging for people with disabilities to access (see here to start digging into that: http://webaim.org/techniques/javascript/).

Thing is, if you have a fallback solution, Google could still access those. However, Google may still be able to access those pages with JavaScript as Google can execute some forms of JavaScript. Given that, the more appropriate solution would be to use the robots.txt file. You mentioned, though, that the command you put in didn't seem to work since Google kept indexing those pages. Couple questions:

First, did Google index those pages after the change or had those pages been indexed prior to the robots.txt change? Things take time, so I'm wondering if you didn't give them enough time to adjust.

The other question would be whether or not you tested the robots.txt file in Google Webmaster Tools? That just gives you an extra verification that it should work.

Also, you mentioned something interesting about the Vehicle Detail pages: "these pages are not meant for visitors to navigate to directly!" Given that is the case, is it possible for your developers to add some sort of server-side check to see if people are accessing the detail pages from the listing pages?

For instance, on some sites I've worked a cookie is set when you've reached the listing page that says "this person is okay to reach the detail page" and then the visitor can only reach the detail page if that cookie is set. Without that cookie, the visitor is redirected back to a listing page. Not sure how exactly that would work on your site, but it might be a way to keep visitors who find those pages in a Google search result from seeing the incorrectly styled page.

I hope that helps. Like I said, feel free to email me or private message me if you'd like me to take a look at your site or chat with you about more particulars.

Thanks!

Hey Dana,

I've had this problem with Safari being slower as well. In fact, I just checked a handful of sites in Google Analytics and Safari is almost consistently the slowest browser. One question to look into is whether or not mobile traffic on Safari is slow. For the sites I'm looking at and for the sites I've worked on in the past, the culprit is usually Safari's desktop browser. Phones are usually within normal loading parameters (by phone standards).

Unfortunately, in most cases, the sluggish performance on Safari is due to Safari's DNS prefetching. Generally, prefetching does the opposite, but apparently can slow you down in Safari. You can read more about that here. http://macs.about.com/od/MacTroubleshootingTips/qt/Troubleshooting-Safari-Slow-Page-Loads-Caused-By-Dns-Prefetching.htm and this is another good one http://computers.tutsplus.com/tutorials/how-to-fix-slow-and-non-loading-webpages-in-safari--mac-51338 (Lots more if you Google Safari prefetching.)

I do think it is worth it to try to fix it. Here's what I would do, if I were you. Feel free to PM me if you want more help...

The way you fix this is on the client's machine, at least according to those articles (and others I've found while Google'ing around just now). However, you can try these steps out on your computer (if you've got a Mac) and see if you have an improved performance in Safari. It shouldn't take that long to test, and is probably worth it given the huge loss you are seeing due to this issue.

The next question is, well, how do I do this for everybody? The answer is to ask your visitors to use Chrome! No, seriously, one that worked for me in the past, was to detect the Safari user agent and load a slightly different version of the page that required fewer requests, meaning there will be less for Safari to prefetch. For instance, you might get rid of JavaScript, images, etc. that isn't essential. Obviously, keep the core content so that the pages are basically the same (kind of like you would for mobile detection).

This is obviously much more complicated to setup as it requires adjustments to the design as well as the code structure. But, generally simpler than trying to redo your entire website.

Before you make any such changes, it might be worth running a simple test on a key landing page that gets a lot of visits from Safari. Remove what elements you can for all user agents and see if this changes anything substantially with the site speed in Safari.

I hope that helps. Thanks,
Matthew