"Google-selected canonical different to user-declared" - issues
-
Hi Moz!
We are having issues on a number of our international sites where Google is choosing our page 2 of a category as the canonical over page 1. Example; https://www.yoursclothing.de/kleider-grosse-groessen (Image attached).
We currently use infinite loading, however when javascript is disabled we have a text link to page 2 which is done via a query string of '?filter=true&view=X&categoryid=X&page=2'
Page 2 is blocked via robots.txt and has a canonical pointing at page 1.
Due to Google selecting page 2 as the canonical, the page is no longer ranking. For the main keyphrase a subcategory page is ranking poorly.
-
Sounds like you had the best of intentions by giving a non-JS fallback but that it came back to bite you
By the way, this gives evidence to something else that I'm always, always banging on about - Google 'can' render JS and do headless browser renders of a web-page when crawling, but they don't do this for everyone and they don't do it all the time (even for sites large enough to warrant such increased crawl resources). Rendered crawling is like 10x slower than basic source code scraping, and Google's mission is to index the web. Obviously they're not going to take a 10x efficiency hit on their MO for just anyone
Sorry about that, needed to get it off my chest as people are always linking articles saying "LOOK! Google can do JS crawling now we don't have to make sure our non-modified source code is solid any more". YES YOU DO - INTERNET
Ok done now. Let's focus on the query at hand
So you have this lovely page here which you have quoted: https://www.yoursclothing.de/kleider-grosse-groessen
It looks like this:
https://d.pr/i/QVNfKR.png (screenshot)
And you can scroll it down, and it infinitely loads - and you only see the bottom of the results (with no page changing button) when results run out, like this:
https://d.pr/i/XECK5Q.png (screenshot)
But when JS is disabled (or if you're fast like some kind of ninja cat, and you scroll down to the bottom of the page and find the button before the infinite load modifies the page-contents... but no mainly, just when JS is disabled) - then you get this button here:
https://d.pr/i/4Y9T9Y.png (screenshot)
... and when you click the button you end up on another page like this one:
https://www.yoursclothing.de/kleider-grosse-groessen?filter=true&view=32&categoryid=3440&page=2
... where you see "&page=2" at the end there, which is the parameter modifier which changes the active page of contents
Google are sometimes choosing the sub-pages of results as canonical when you guys don't want them to do that. You want to know why, what you have done isn't really working and what you could do instead. Got it
IMPORTANT Disclaimer: Google decides to rank pages for a number of reasons. If Google really does feel that sometimes, sub-pages of your results are 'better' (maybe they have better products on some of the paginated URLs, a better mix of products or products which fit Google's idea of fair pricing better than the default feed...) - there is no guarantee that 'correcting' this 'error' will result in the same rankings you have now. I just want to be 100% clear on that point, you might even lose some rankings if Google is really decided. They have told you, they are overriding your choice and usually there's some kind of reason on that. Sometimes it's a 'just past the post' decision where you can correct them and get basically the same rankings on other pages, other times you can lose rankings or they just won't shift it
Still with me? Ok let's look at what you did here:
-
On the page 2 (and page 3, and however many paginated URLs there are) you have a canonical tag pointing to the parent
-
And you have blocked the paginated URLs in robots.txt
I need to start by querying the fact that you say the page 2s (and assumedly other sub pages, like page 3s - e.g: https://www.yoursclothing.de/kleider-grosse-groessen?filter=true&view=32&categoryid=3440&page=3) - are blocked in robots.txt
DeepCrawl's indexation plugin doesn't see them as blocked:
https://d.pr/i/1cRShK.png (screenshot)
It says about the canonical tag, but it says nothing about the robots.txt at all!
So lets look at your robots.txt file:
https://www.yoursclothing.de/robots.txt
https://d.pr/i/YbyEGl.png (screenshot)
Nothing under # BlockSecureAreas handles pagination
But then under # NoIndex we have this entry:
Disallow: /filter=true
That _should _handle it, as pagination never occurs without a filter being applied (at least as far as I can see)
Indeed using this tool that I like, if I just paste in only the relevant parts:
https://d.pr/i/TVafTL.png (screenshot)
**We can see that the block is effective **(so DeepCrawl, your Chrome tool is probably wrong somehow - maybe they will see this new link, read and fix it!)
I did notice that there's some weird, unrequired indentation in your robots.txt file. Could that cause problems for Google? Could it, at the least - make Google think "well if there's syntax errors in here, maybe it's not worth obeying as it's probably wrong" - quite possibly
In my opinion that's not likely to be part of it
So if it's not that, then what!?
Well it could be that you're using robots.txt in the wrong capacity. Robots.txt _doesn't _stop Google from indexing web pages or tell them not to index web-pages (which is why it's funny that you have commented with "# NoIndex" - that's not what robots.txt does!)
Robots.txt dissuades Google from 'crawling' (but not indexing) a URL. If they can find signals from around the web (maybe backlinks) or if they believe the content on the URL is better via other means, they can (and will) still index a URL without necessarily crawling it. Robots.txt does not do, what Meta no-index does (which can be fired through the HTTP header, or via HTML)
Also, riddle me this if you will. If Google isn't allowed to crawl your URLs any more, how will it continue to find your canonical tags and find any new no-index tags? Why give Google a directive (canonical tags) on a URL which Google isn't allowed to crawl, and thus they will never see the directive? Sounds backwards to me
My proposed steps:
-
Read, understand and make your own decision on the "disclaimer" I wrote up earlier in this very post
-
If you still want to go ahead, enact the following (otherwise don't!)
-
Remove the robots.txt block so Google can crawl those URLs, or if that rule covers more than just the paginated URLs - leave it in place but add an exclusion for the paginated URLs so they may be crawled
-
Leave all the canonical tags on, good work. Maybe supplement these with a 'no-index' directive which would tell Google not to index those pages (there is no guarantee the canonical URL will replace the no-indexed URL, but you can try your luck - read the disclaimer)
-
Maybe serve status code 410, only to Googlebot (user-agent) when it visits the paginated URLs specifically - to try and encourage Google to think of those URLs as gone. Leave the contents alone, otherwise it's cloaking. Serve the same content to Google and users, but serve googlebot a 410 (gone) status
-
Before enacting the super-aggressive 410 stance, give Google plenty of time to swallow the new "no-index" tags on paginated URLs which weren't there before. A 410 whilst powerful, may cause these not to be read - so do give Google time (a few weeks IMO)
-
If you do adopt the 410 stance, one down-side will be that Google will think your JS fallback is a broken link and this will appear in Google Search Console. To make this less severe (though it probably still will happen), add no-follow directives to the pagination JS-fallback link / button where it appears
-
Once Google seems to have swallowed your wishes and seems to have removed most of these URLs from their index, THEN put the robots.txt block for paginated URLs back on (so it won't all happen again in the future)
-
Try removing the weird indentation formatting from your robots.txt file
-
Smile
Well, that's it from me. Thanks for this one, it was pretty interesting
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Titles and User Intent
Hi all, Just a question re: Titles. Does adding terms like "buy online" to the title for ecommerce websites change the way Google ranks you? For example: Blue Flowers | Flower Palace Buy Blue Flowers Online | Flower Palace Would the second title appear higher for people in the buying phase of the sales funnel, and lower for people in the research phase of the buying funnel? Or will it not matter?
On-Page Optimization | | Onlineorders0 -
Keyword Stuffing Issues
Hi Folks, At this URL https://purplegriffon.com/courses/project-management/prince2 as an example, you can see that I am outputting all course dates in a table. Each table row contains information regarding the specific course such as start date, duration, location but also course title. In the course title it will obviously contain the keyword, in this example that is 'prince2'. So my question is this. As the MOZ on-page grader indicates that I am keyword stuffing (even though I receive an A grade) will Google be clever enough to know that I am displaying a table of events for PRINCE2? Or will it class the page as containing too many keywords? Each event links to a specific event page for that event, so is unique. I am interested in hearing the thoughts of the community on this. Thanks. Regards Gareth
On-Page Optimization | | PurpleGriffon0 -
What makes a "perfectly optimized page" in 2013?
Hi all, I was re-reading this blog http://moz.com/blog/perfecting-keyword-targeting-on-page-optimization and wondered how relevant you guys thought this post still was? Moz link directly to it from their 'learn' page http://moz.com/learn/seo so I'm interpreting that to mean it is still accurate and as current as it can be? What else would you add to it? Thanks,
On-Page Optimization | | James-Distinction
James0 -
Why isn't our site being shown on the first page of Google for a query using the exact domain, when its pages are indeed indexed by Google
When I type our domain.com as a query into Google, I only see one of our pages on the homepage, and it's in 4th position. It seems though, that all pages of the site are indexed by google when I type in the query "site:domain.com". There was an issue at the site launch, where the robots.txt file was left active for around two weeks. Would this have been responsible for the fact that another domain ranks #1 when we type in our own domain? It has been around a couple of months now since the site was launched. Thanks in advance.
On-Page Optimization | | featherseo0 -
How do I address "Critical Factors: Accessible to Engines"?
Hello,I am going thru the on-page report card produced by SEOMOZ and am stumped as to how to address the first critical factor. It looks like the correct meta tag to get search engines to index the site is at the bottom of the header. And as far as I know, which isn't much, the site returns the HTTP code 200 when I refresh.I am new at this, so please let me know if you have some specific solutions. I am using IWeb and the IWeb SEO Tool to make meta code improvements. I have pasted the head code for my website (www.grass2greens.com) below. Thanks in advance!<html lang="en" xml:lang="en" xmlns="http://www.w3.org/1999/xhtml"><head><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="iWeb 3.0.4" name="Generator"><meta content="local-build-20120619" name="iWeb-Build"><meta content="IE=EmulateIE7" http-equiv="X-UA-Compatible"><meta content="width=880" name="viewport"><title>Grass to Greens: Asheville Edible Landscapingtitle><link href="Grass_to_Greens__Asheville_Edible_Landscaping_files/Grass_to_Greens__Asheville_Edible_Landscaping.css" media="screen,print" type="text/css" rel="stylesheet"><style type="text/css"><script type="text/javascript" async="" src="http://www.google-analytics.com/ga.js"><script type="text/javascript" async="" src="http://www.google-analytics.com/ga.js"><script src="Scripts/iWebSite.js" type="text/javascript"><script src="Scripts/iWebImage.js" type="text/javascript"><script src="Scripts/iWebMediaGrid.js" type="text/javascript"><script src="Scripts/Widgets/SharedResources/WidgetCommon.js" type="text/javascript"><script src="Scripts/Widgets/HTMLRegion/Paste.js" type="text/javascript"><script src="Grass_to_Greens__Asheville_Edible_Landscaping_files/Grass_to_Greens__Asheville_Edible_Landscaping.js" type="text/javascript"><script type="text/javascript"><meta content="Grass to Greens offers a range of edible landscape design, consultation, installation, and maintenance services. Free Consultations! We specialize in beautiful and useful vegetable gardens, season extension, tree work, orchards and food forests, stone work, fencing, and rain water catchment. Grass to Greens is an edible landscaping company committed to creating food security and fostering social justice through urban agriculture in the Asheville area. " name="description"><meta content="Landscaping Asheville Edible Gardens" name="keywords"><meta content="follow,index" name="robots"><link rel="stylesheet" type="text/css" href="Grass_to_Greens__Asheville_Edible_Landscaping_files/Grass_to_Greens__Asheville_Edible_LandscapingMoz.css">head> Grass to Greens: Asheville Edible Landscaping
On-Page Optimization | | dcaudio0 -
Tags creating duplicated content issue?
Hello i believe a lot of us use tags in our blogs as a way to categorize content and make it easy searchable but this usually (at lease in my case) cause duplicate content creation. For example, if one article has 2 tags like "SEO" & "Marketing", then this article will be visible and listed in 2 urls inside the blog like this domain.com/blog/seo and domain.com/blog/marketing In case of a blog with 300+ posts and dozens of different tags this is creating a huge issue. My question is 1. Is this really bad? 2. If yes how to fix it without removing tags?
On-Page Optimization | | Lakiscy0 -
Rel Canonical
I will be quick and to the point. I am clearly a novice. I received a notice on my seomoz account that I had 12 pages with a Rel Canonical issue. It seems serious. Can this be a quick fix? Any thoughts? Below is the site address. petbarnpdx.com Thanks in advance for any input on this! Dave
On-Page Optimization | | APICDA0 -
Has anyone noticed a big delta between Google and Bing rankings? For example, we rank favorably in Google, but not so favorably in Bing. Are there different tactics I should use to rank better in Bing?
An example is in Google, we currently rank #1 & #2 for "yoga pants" for Athleta and Old Navy. In Bing, I'm on page 2. Any thoughts here?
On-Page Optimization | | kpr0