Welcome to the Q&A Forum

TylerReardon

Hey Katherine,

If you could share the URL of your home page, I could likely tell you what the issue is but if you don't feel comfortable doing that, here are a few suggestions.

1. Install the "Link Redirect Trace" Chrome extension. An awesome extension that gives you the HTTP headers on every URL in real time.
2. Check for some pretty standard domain redirect issues such as:
-http://example.com to http://www.example.com
-http://example.com to https://example.com
-https://example.com to https://www.example.com

The http to https redirect is very common due to old links, the habit of typing http, and the fact that browsers still default to http unless you specify otherwise.

For instance, if you type example.com in the nav bar, your browser will assume you mean http://example.com and try that, at which time you'll be redirected to the https version.

The .htaccess file (if you're on apache) can be a good place to fix the issue once you know what is happening. It's also worth noting that redirects on your home page are not necessarily a bad thing as long as they're one of the aforementioned examples and they're 301 redirects.

I hope this helps. Let me know if you have any questions!
-Tyler

TylerReardon

That's awesome! Glad to know there's a bulk tool out there!

TylerReardon

Hey Daniel,

If you want to verify that a user-agent is actually Googlebot, you'll want to use a log file analysis tool to aggregate all of the IP addresses associated with Googlebot. Once you have a list of IP addresses, you can perform a reverse DNS lookup to verify whether the IP addresses are actually associated with Googlebot or not.

If you're on windows/pc these steps should work:
https://www.serverintellect.com/support/dns/reverse-dns/

If you're on mac try these steps:
1. open Terminal
2. type "host" + ip address
for example: "host 66.249.66.1"
3. hit enter
4. view results. For example: "1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com"

If the results are from Google.com or Googlebot.com, you can be sure it's actually Google crawling your site. Unfortunately, I don't know of any faster ways to achieve these results. I'm sure there's a tool out there, I just haven't found it yet.

This might also be a good resource for you: https://support.google.com/webmasters/answer/80553?hl=en

Good luck!
-Tyler

TylerReardon

Yes exactly. Even though the pages don't exist to the user, they still technically exist. If I were you, I'd take a very deep look at pagination on your site. If this is happening at scale, then fixing it could be a major improvement to your site. I took a look and it seems to be happening on all your top-level category pages like Chairs, Office Furniture, Shelving & Racking, etc.

These paginated pages are essentially a bunch of duplicate pages of your main category pages, each with a self-referencing canonical (which is the proper way to set up pagination). So Google could be extremely confused about which one to rank. In most cases, Google will rank page 1 because the use of rel="next"/rel="prev" is essentially telling Google that page 1 is the canonical version. However, you're still opening yourself up to the possibility of Google crawling all of these duplicate pages which is a huge waste on your crawl budget.

Hope that helps!

TylerReardon

Yes, 301 redirections are still respected and should pass all "page rank" (if page rank still exists). If you're doing a site migration and plan on using 301 redirects for a high volume of URLs, it's not uncommon to see an initial drop in rankings as Google re-evaluates the site. Don't panic if that happens, you should return to normal as the only thing that changed is the URL.

Be very careful as 301 redirects can be costly if executed incorrectly. Make sure to check the redirects with an HTTP status code checker.

TylerReardon

If you view your source code, you'll notice you are actually using rel="next" and rel="prev" on the main category page (https://www.key.co.uk/en/key/chairs). This is why you (and most likely Googlebot as well) are crawling these paginated pages. Even though you don't have links to the paginated pages on the main category page, they still exist and you're giving crawlers the directive (rel next / rel prev) to crawl them.

If you remove rel="next" on the category home page, that should help but you should really remove rel="next" and rel="prev" on the paginated pages as well. Unless you do that, Google will still find them and crawl them because they're aware these pages exist and they're likely indexed.

Here's a great resource on understanding pagination as well as the correct use of rel="next" and rel="prev" from Maile Ohye at Google: https://www.youtube.com/watch?v=njn8uXTWiGg

Hope this helps!
Cheers!
-Tyler

a7jXKhU

TylerReardon