My automated build system is creating a duplicate website
-
Because of the tools my company is using for CI/CD (A CI/CD pipeline helps you automate steps in your software delivery process, such as initiating code builds, running automated tests, and deploying to a staging or production environment.) an extra URL is generated. The canonical for the generated site is that of our main website, but other than that it is the same website.
- Could this new URL compete with our website?
- Will Google count it against us since it is the same content BUT with canonical (it is not noindex-ed)?
- Does it matter?
- Surely others are using this method?
Answers/thoughts will be greatly appreciated. Thank you.
-
Do you have any control over the CI/CD pipeline URL?
If you control the domain enough so that you can be one to have validated and searched console them by all means. But it does not seem like you have the ability to control domain?
my correct?
https://support.google.com/webmasters/answer/7440203?hl=en
If the domain is 3ed party domain then you must trust the third-party or if you control the domain of pages which links or third-party domain URLs are embedded on you can add noindex nofollow
https://www.deepcrawl.com/blog/best-practice/noindex-disallow-nofollow/
I hope that helps,
Tom
-
Unfortunately, since URL is generated from the original site, I cannot change the robots.txt. It uses the same one as the main site. That would exclude adding a noindex meta tag, as well. Any other ideas?
Is there a way to add the duplicate URL to search console & tell google not to crawl?
Thank you.
-
I understand using CI cool
i agree get the bad content being made by CI blocked ASAP
“have an extra URL is generated. The canonical for the generated site is that of our main website, but other than that it is the same website.”
but it’s not the same content being made that will hurt you unless you’re pointing the canonicals to a similar page (get the automated content off your domain)
Remember to add using self pointing canonicals on the good pages you want to be indexed by Google or Search Engines
Hope this is of help,
Tom
-
To answer your questions:
- Technically it could compete with your current site as it's on its own domain, in reality, it's unlikely as you're canonicalizing the pages back to its original and making sure that the content itself through that way is attributed to your original site.
- What I would recommend is excluding the CI/CD site from the engines, through a robots.txt or a similar technique. That way you're making sure that the staging site itself isn't being crawled at all. In the end, I'd say there's very little upside of having that be the case currently.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a tool to check the SEO performance of articles from different websites?
Is there a tool to check the SEO performance of articles(like 100+ articles) from different websites in one place? I am looking for a tool where I can put several URLs of the articles we have published in bulk for many of our clients, and it can show me how those articles are performing. This way, we can check the performance of these articles at any time; there is no need to add URLs every time.
Reporting & Analytics | | Foxxr0 -
How do I create a segment that shows me all pages using a certain keyword? But nothing that doesn't have that keyword?
There must be an easy answer to this, but I can't seem to find it. All I want to do is create a segment in Google Analytics that shows all pages and search strings with "orthopaedics" in the title, with pageviews, uniques etc. If I simply navigate to "All Pages" in Google Analytics and then click Advanced Filters and do an Include Page Contains "orthopaedics" it works just fine. (See attached Screen Shot) But when I try to recreate this as a segment, it pulls in all other pages the users visited before arriving on the orthopaedics page I want to include, which I don't want. I can manually exclude each URL I don't want, but this is tedious and I feel there must be a simpler method I'm just missing. At the end of the day, I'm trying to create a list of every page and dynamically created query string that includes the word "orthopaedics" to say doctor X, your orthopaedics section generated X views, and here's a list of the pages. Mm6YTKa
Reporting & Analytics | | Patrick_at_Nebraska_Medicine0 -
How can I stop deadlink checker from visiting my website?
It is messing up my Google Analytics traffic reporting. I can't figure out how to get it to stop. Do I filter it out in GA?
Reporting & Analytics | | pmull0 -
Been stuck on seo duplication issues shopify
hey there we have been working on some of our webshops and recently started with analytics/moz,but we have basicly hit a brick wall when it comes to www.krawattenwelt.de since we have had 5k high priority issues (duplicate content) and 20k medium priority issues now i have tried a large amount of solutions regarding the duplicate content issues but it didnt work so we basicly reverted it back to for now and i have the feeling i am really running out of options is there anyone who has an idea on how to do this? duplicate content issues are as follows example:http://krawattenwelt.de/collections/budget-9-15 issues with:http://krawattenwelt.de/collections/budget-9-15/modell_normal and with:http://krawattenwelt.de/collections/budget-9-15/modell_normal?page=1
Reporting & Analytics | | WebMaster2050 -
Creating Meta Tags for a Web Hosting company
My client has recently had his site rewritten by a company which supplies a website/hosting/seo service. The client still wants to use me for establishing his site locally and for SEM. My problem is that the code is not accessible to me. They have set up GA tracking which feeds into the reports dashboard. There is no way of actually access the raw data. When I asked for access to the GA account I got this message from bOline solutions: "Your SEO consultant will need to create a new GA account and then add the meta tag to the reports tab of your site" Because I am still very much learning GA I do not know what meta tag they are referring too. I'm therefore stuck as to how to create it. I've started learning about "Google Tag Manager" and am working through "Digital Analytics Fundementals", I've a feeling this is just a question of terminology - can anyone more experience in the subject help me?
Reporting & Analytics | | catherine-2793880 -
Pages with Duplicate Page Content
Hi Just started use the Moz and got an analytics report today! There about 104 duplicate pages apparently, the problem is that they are not duplicates, but just the way the page has been listed with a description! The site is an Opencart and every page as got the name of the site followed by the product name for the page! How do you correct this issue?? Thank for your help
Reporting & Analytics | | DRSMPR1 -
Moz Crawler suddenly reporting 1000s of duplicates (BE.net)
In the last 3-4 days we've had several thousand 'duplicate content' warnings appear in our crawl report, 99% of them related to our on-site blog. The blog is BlogEngine.Net, but the pages simply don't exist. The majority seem to be Roger trying quasi-random URLs like:
Reporting & Analytics | | Progauto
/?page=410 /?page=151 Etc. etc. The blog will present content for these requests, but it is of course the same empty page since there's only unique content for up to /?Page=10 or so. Two questions: 1. Did something change recently? These blogs have been up for months, and this problem has only come up this week. Did Roger change to become more aggressive lately? 2. Suggested remediation? On one of the blogs I've put no-index no-follow for any page that has a /?page querystring, and we'll see what effect that has come next crawl next week. However, I'm not sure this will work as per: http://moz.com/community/q/functionality-of-seomoz-crawl-page-reports Anyone else had dynamic blogs suddenly blossom into thousands of duplicate content warnings? Google (rightly) ignores these pages completely.0 -
Duplicate content and ways to deal with it.
Problem I queried back a year for the portal and we can see below that the SEO juice is split between the upper and lowercase. You can see the issue in the attached images. http://i.imgur.com/OXnPp.png Solutions: 1) Quick: Change the link on the pages above to be lowercase 2) Use canonical link tag http://www.seomoz.org/blog/canonical-url-tag-the-most-important-advancement-in-seo-practices-since-sitemaps The tag is part of the HTML header on a web page, the same section you'd find the Title attribute and Meta Description tag. In fact, this tag isn't new, but like nofollow, simply uses a new rel parameter. For example: http://www.darden.virginia.edu/MBA" /> ''This would tell Yahoo!, Live & Google that the page in question should be treated as though it were a copy of the URL http://www.darden.virginia.edu/MBA and that all of the link & content metrics the engines apply should technically flow back to that URL.'' 3) See if there is any Google Analytics filters at the site level I can apply. I will check into this and get back to you. What do you all think?????? OXnPp voJdp.png OXnPp.png
Reporting & Analytics | | Darden0