Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
Completely agreed with Dana on this. I recently helped a startup fixing this problem by setting a redirect rule in their nginx configuration to redirect all the URLs with trailing slash to desired version. PFB the required rule if you're using nginx:
rewrite ^/(.*)/$ /$1 permanent;
If you're on LAMP/MAMP/WAMP tech stack, find the apace equivalent of this rule and place it in your apache config file. Hope that helps!
Trying to upload multiple keywords at a time with their different Tags. But here i can upload the keyword one by one also i am not able to associate tags with the keyword.
Sure, please share an example URL to help me suggest you on this.
Hi David,
If you're planning to switch to the new domain, I would recommend you to not close the existing domain but redirect all the pages to their equivalent pages on the new domain. Please note, relevance is the most important factor here from SEO perspective. Gradually, all your indexed pages will be replaced by their corresponding new pages (ranking might not be the same though).
Also, keep an eye on search console for the domain A as well. This will give you an idea on how many pages are still breaking and then you can quickly fix them or, redirect to their corresponding new pages.
P.S Of course you'll loose your domain authority here, but if its a business decision, this is how you can handle it the best way I believe.
Hope this helps!
Hi,
This is a problem related to site's architecture i.e you need to expose only those pages to bots that you want to and while doing this, make sure you're not ending up with keyword cannibalization wherein you have more than one pages targeting a single keyword like in this situation.
So, its always better to take this call in the start whether you want to expose tag pages to bots or not? They might be serving exactly same content as your category pages for some keywords. As a result, you might end up with duplicate content and keyword cannibalization situations.
Short answer to your question, yes canonical should be fine here if the content is almost the same and targeting the same keyword. I would recommend you to not do this for individual pages, rather find a pattern and do it for all to avoid the same problem again.
Hope this helps!
Sure. Thank you so much for confirming the same.
For the first time today, my recommended solution for a problem was supported by you. I'm really happy about it. "Moving in the right direction" feeling 
Hi Sir Alan,
Thank you so much for your response. I agree, actually I also recommended the same solution of creating new brand pages and then linking them from the product pages but they (product team) wants to show brand description on product pages itself.
Could you please recommend any other solution here? If there can be another way to implement this. Or else, we might show a few words of description there along with a link to brand page.
Thanks Don. I meant this only.
Hi Don,
Using these generic blocks for multiple products is an awesome idea, but don't you think it'll end up creating almost duplicate content section on all these product pages?
Does bots consider duplicate content across the page, or section wise?
Hi Don,
Thank you so much for your well detailed answer, really appreciate that.
My question was a little different here. Yes, we're planning to have unique description for the products and recommended products etc. to make the overall content unique. My question was about a specific section on the page where we're planning to show the brand description, which would be exactly same for all the products of a specific brand.
So, how does google (and other bots) consider duplicate content for a specific section, and not the entire page?
Hi Mozzers,
Hope you're doing good. I have a content placement related question.
Assume, I have 1000 products of brand A, 1000 of brand B, and so on. Now, if I want to put brand specific 200-words description on each of these product pages. I'm creating duplicate content across the site by putting absolutely same brand description on these product pages i.e brand A description on first 1000 pages, brand B description on next 1000 products and so on.
Looking for an expert advice around placement of content here i.e how can I add brand description on product pages and avoid duplicate content penalty? Any help?
Hi Martijn,
Yeah, not planning to block them from robots.txt of course. By blocking, I meant reducing the crawl rate to ZERO temporarily to make sure we're not creating any URL related confusions for bots.
But, this might not be a good solution for our customers as customer might be redirecting to /new-url for the first hit, which might give him an error for in the next session.
Hi Martijn,
Thank you so much for sharing your thoughts on this, really appreciate that.
But, the problem here is, we're planning to launch some of the new urls which aren't backward compatible. Hence, exposing them to bots isn't a good idea, they won't work for old website.
Also, if I 301 redirect the /old-url to /new-url, would need to redirect it back to /old-url if hit goes to old website. This might confuse bots.
Btw, number of URLs affected by this is, the almost whole website i.e a very large number of indexed pages.
Hi Mozzers,
Hope you're doing good. Well, we have a website, up and running for a decent tenure with millions of pages indexed in search engines. We're planning to go live with a new version of it i.e a new experience for our users, some changes in site architecture which includes change in URL structure for existing URLs and introduction of some new URLs as well.
Now, my question is, what's the best way to do a A/B test with the new version?
We can't launch it for a part of users (say, we'll make it live for 50% of the users, an remaining 50% of the users will see old/existing site only) because the URL structure is changed now and bots will get confused if they start landing on different versions.
Will this work if I reduce crawl rate to ZERO during this A/B tenure? How will this impact us from SEO perspective? How will those old to new 301 URL redirects will affect our users?
Have you ever faced/handled this kind of scenario? If yes, please share how you handled this along with the impact. If this is something new to you, would love to know your recommendations before taking the final call on this.
Note: We're taking care of all existing URLs, properly 301 redirecting them to their newer versions but there are some new URLs which are supported only on newer version (architectural changes I mentioned above), and these URLs aren't backward compatible, can't redirect them to a valid URL on old version.
Hey,
Stone Temple Consulting recently published this article, that's exactly answering your question. Hope this helps!
Hi Neil,
Here's the answer to your question. Yes, they do index, and index well.
You can check all the indexed pages for your domain here: https://www.google.co.in/search?q=site:realhomes.inspirythemes.biz/listing.
Click on the drop-down button and click on "Cached" to see its google cached version. Then, click on "Text-only version" to see what exactly bots are able to see on your page.
Hope this helps!
Hi Jay,
I would suggest you to buy the .com domain if its available. Using a country specific domain to target international market is not a good idea. If .com isn't available, you can think about buying country specific domains to target various countries.
https://moz.com/learn/seo/international-seo can help you understand more on this. Hope this helps!
Could you please share the URL here? Would like to have a look at the way you've implemented this.
Hi,
The best way to handle this kind of scenario is by implementing pagination wisely. Even if you don't have a view all page for your content, may be you're doing lazy loading or something similar to show the entire content to your users but bots' HTML snapshot misses all this content. Hence, its better to support pagination and expose them to bots using proper pagination.
All you need to implement is a doubly linked list kind of structure where your first page's next pointer points to second page and it doesn't have prev pointer; second page's prev pointer points to first page and its next pointer points to the 3rd page; similarly last page doesn't have next pointer but its prev pointer points to the second last page. You can refer this guide for implementing the same. This site can be referred to understand how exactly to implement this.
Yes, its really trivial to implement this, you might need a dev's help though if yoast doesn't support this. Hope this helps!
Feel free to write here if you need any further assistance on this. Cheers!
Thanks for sharing your experience and this great article Cyrus, really useful stuff.
What's your observation on post migration traffic trends? Does it makes sense to migrate to https for SEO perspective only?
Hi Dirk,
Good observation, I missed the canonical part somehow. So, google is indexing the canonical URLs here which doesn't have /blog/ in it and that's the problem. Have a look at the indexed page for this particular instance here. Non /blog/ instance is indexed, which will take you to its /blog/ version with wrong canonical URL.
Solution: Either remove the canonical URLs on these pages to point them to the current page itself. And yeah! As rightly mentioned by Dirk, do a proper /blog/ page linking from the blog page and other pages from where you're linking these articles.
Hi Rob Morgan,
You've talked about 2 issues here, please find my views on them:
"Since launching my new website design - www.advanced-driving.co.uk I am not convinced Google is seeing all the content on the page. I took a long extract of text and did a search on Google and nothing was found. Also although in the search results for "advanced driving course" I can see the new title tag, the snippet isn't showing"
=> Here, you can check the cached version of any given page by googling "cache:<url-you-want-to-check>" and then clicking on "Text Version".</url-you-want-to-check>
As a scroll down I can see the URL changes i.e:
www.advanced-driving.co.uk
then: http://www.advanced-driving.co.uk/#da-page_in_widget-3
then: http://www.advanced-driving.co.uk/#da-page_in_widget-4
then: http://www.advanced-driving.co.uk/#da-page_in_widget-5
=> This is absolutely fine as far as canonical URL is there, and its there in your case. So, good to go
Hope this helps!
Hi Jason,
I had a word with a friend who handled the similar project for a large organisation a month back and what they observed is, the traffic went down initially for sometime and picked up the flow later on.
Hi Happy SEO,
Well, the robots.txt looks find here. Could you try to fetch any of the blog page/post as google in the search console and share the screenshot here?
Also, to cross check the robots.txt (which looks fine though), you have robots.txt tester in search console where you can put any blog page/post to check if bots can crawl it. Please share a screenshot of that as well.
On a separate note, the sitemap.xml link mentioned in the robots.txt (http://www.keukensduitsland.nl/sitemap.xml) is broken. Fix that as well.
Hi Jamie Booker,
Interesting point. I would like to share my points to handle it from UX and SEO perspective neatly. So, here's my understanding about your product: you have multiple product pages and their variations, say you have a product "My Product" which has 2 variations "Variation A" and "Variation B" (variation can be based on color, type etc. attributes).
Here, you can itemise these product variations and consider the following solution to handle it:
Variation pages would be available at www.example.com/my-product/variation-a and www.example.com/my-product/variation-b respectively for "Variation A" and "Variation B" of the product "My Product".
Make sure you have the canonical URL without the variation id/title set for your variation pages. For instance,
for these variation pages.
Mark one of these variations as your default product which will be available at www.example.com/my-product. For example, www.example.com/my-product will display the same content that www.example.com/my-product/variation-a will display.
This way you'll have one single URL for a product page for bots which is www.example.com/my-product and all other variation pages exist with variation identifier which will resolve customer's experience point. Also, as we're exposing only single page to bots in this case, we're neatly able to handle duplicate content penalty issue as well.
You can refer the following 2 variation URLs for the same:
Hope this helps!
Hi Happy SEO,
Could you please share the blog URL here? Sounds like an interesting issue and would love to give a try to help you with this
Thank you so much Sir Alan. Really appreciate your reply
Oh wow! Will connect with you on Twitter to understand about the same which can help me plan it better. Hope you won't mind sharing the way your architected your internal tracking tool
Hi Linda,
Yeah! High time to start exploring GA annotations. Needless to say, will definitely post here once I'll be able to find/build a good solution for the same
Hi Cesar Bielich,
Thank you so much for the well descriptive explanation, will start exploring GA annotations right away.
Yes, I can code and planning to work on internal analytics system to track these granular pieces but it'll take time to implement such powerful system when you've 10 million + pages and hence, its not a P0 right now. We have integrated GTM as well, and tracking some of these values to some extent. But, as you correctly mentioned that none of these things can be directly mapped to any increase/decrease in organic traffic, I should definitely think about prioritizing my project to understand the correlation between my changes and the organic traffic which can be an awesome asset to understand these things.
Hi Mozzers,
I was just wondering if there's a way we can map our on-page SEO changes with the increase/decrease in organic traffic. For instance, I introduced brand pages' link the product page breadcrumbs and suddenly organic traffic for my brand pages increase from X to 2X in 1 couple of weeks. Now, this can be because of this breadcrumb change purely or because of some algorithm update or may be, bots started finding the content interesting and hence, started ranking them up (in case the brand pages were launched recently).
So, you can't say which change should be mapped to what increase/decrease in organic traffic. Or, is there a way to map this?
Hi,
IMO, don't remove/de-index/404 the pages which are not your traffic drivers today, they might bring some traffic for you tomorrow. I assume, these pages aren't hurting (going-to-hurt) you i.e they are not duplicate content candidates or something.
Of course you can promote whatever pages you want, like you want to promote those 2k traffic drivers in your case.
P.S Taking down a page and showing 404 is not a good practice in general. If you're permanently closing a page and don't want to 301/302 redirect it, then handle it using 410 instead of 404 here. You can check more about http status codes here: https://moz.com/learn/seo/http-status-codes
Well, these aren't "useless" links. After all, they're linking your categories/sub-categories etc. and should be followed by bots even if a HTML snapshot of any page captures 2-3 follow links (from flyout/menu navigation, breadcrumbs etc.) of another page.
Hope this helps!
No! Don't "nofollow" them. Why do you want to nofollow them now?
You're not spamming here, inter-linking from flyout-navigation/header/footer/on-page-navigation/breadcrumbs are the natural ways people use for internal-linking, that won't hurt you for sure.
Are you trying to say that you're planning to have multiple URLs for a single product page here? For instance, if you have a product which can be reached from multiple navigation paths, so you want to have those multiple URLs for it?
Like if a product is tagged in category "x" and "x" is a sub-category of category "y", then the number of possible URLs for product page "p" would be
So, here these 2 URLs are candidates of duplicate content penalty and hence, you want to noindex them? Is this what you're trying to explain?
Hi,
Well, following "pyramid" scheme and noindexing pages are two different things altogether. Let's not mix them, its creating confusion actually. So, tell me why do you want to noindex your pages?
Using pyramid scheme and optimizing your site's architecture the best possible way can be done independently.
Hi,
Could you please help me understand your concern here? What do you mean by "noindex, follow all links except 1 from breadcrumbs"?
May be you need to elaborate your concern or share some screenshots to help me understand it.
P.S noindexing a subset of pages is not harmful for sure.
Hi Panagiotis Triantopoulos,
There isn't any hard and fast rule for it. So, go ahead with a different anchor text without a doubt. As far as it's making sense w.r.t the linked article (descriptive enough to talk about the article theme), its good to go.
Hope that answers!
No problem. Do share screenshots of product pages and the URLs (once available) here. Will be able to help you out with this. Fixing is using canonical or meta robots is not a time taking solution to implement in general and hence, can be fixed at the last moment (before going live) as well. So, this can be parked for now.
Hi David,
Thanks for sharing a couple of instances to help me understand the point here. Well, I don't think there is any need of blocking these pages from indexing. You're confused about it just because you don't have much content to show on these pages and the templates is similar and hence, google might consider them as duplicate pages, right?
To resolve this issue and also, to make these pages stronger from organic visibility perspective, you would need to add on-page content and other "cool" features to make them powerful anyways. But, blocking them for bots won't be a good solution I believe.
Btw, if sharing the URLs of the pages is not possible as its in development phase, could you please share the screenshots of the pages here? Would be able to comment on how this should be handled once after having a look at it.
Hi David,
If header and other details are different on these pages, why would you like to set canonical or somehow block these pages from indexing? That should be a candidate for duplicate content penalty I believe.
Could you please share some sample URLs to help me understand the issue you're talking about? I'll try my best to guide to handling this neatly from SEO perspective.
Hi,
I'm working on a website with more than 10 million URLs and hence, couldn't rely on these tools for this. Wrote a script to generate the XML sitemap, it takes a text file with all the URLs and generates the required XML sitemap considering 50,000 URLs per sitemap limit etc. things. Let me know if you have list of URLs with you and interested in generating it at your own (you can ask your development team to integrate this script for auto-generation etc. as well), I'll share the script with you. Cheers!
No my friend, no! I'm saying we'll point the existing staging/testing environment to the production version and will stop using it as staging instead of closing it completely like I mentioned earlier. And, we'll launch a fresh instance for staging/testing use case.
This will help us transferring majority if the link juice of already indexed staging/testing instance.
Hi Eric,
Well, that's a valid point that bots might have considered your staging instances as the main website and hence, this could end up giving you nothing but a face palm.
The solution you suggested is similar to the one I suggested where we are not getting any benefit from the existing instance by removing it or putting noindex everywhere.
My bad! I assumed your staging/testing instance(s) got indexed recently only and are not very powerful from domain & page authority perspective. In fact, being a developer, I should have considered the worst case only
Thanks for pointing out the worst case Eric i.e when your staging/testing instances are decently old and you don't want to loose their SEO values while fixing this issue. And, here'e my proposed solution for it: don't removed the instance, don't even put a noindex everywhere. The better solution would be establishing a 301 redirect bridge from your staging/testing instance to your original website. In this case, ~90% of the link juice that your staging/testing instances have earned, will get passed. Make sure each and every URL of the staging/testing instance is properly 301 redirecting to the original instance.
Hope this helps!
Hey,
I recently faced the same issue when the staging instances got indexed accidentally and we were open for the duplicate content penalty (well, that's not cool). After a decent bit of research, I followed the following steps and got rid of this issue:
This helped me fixing this issue asap. Hope this helps!