Is it possible to deindex old URLs that contain duplicate content?
-
Our client is a recruitment agency and their website used to contain a substantial amount of duplicate content as many of the listed job descriptions were repeated and recycled. As a result, their rankings rarely progress beyond page 2 on Google. Although they have started using more unique content for each listing, it appears that old job listings pages are still indexed so our assumption is that Google is holding down the ranking due to the amount of duplicate content present (one software returned a score of 43% duplicate content across the website).
Looking at other recruitment websites, it appears that they block the actual job listings via the robots.txt file.
Would blocking the job listings page from being indexed either by robots.txt or by a noindex tag reduce the negative impact of the duplicate content, but also remove any link juice coming to those pages?
In addition, expired job listing URLs stay live which is likely to be increasing the overall duplicate content. Would it be worth removing these pages and setting up 404s, given that any links to these pages would be lost? If these pages are removed, is it possible to permanently deindex these URLs?
Any help is greatly appreciated!
-
HI Harry, He should be able to set this up dynamically. Otherwise, yes I do feel sorry for him but then he didn't he didn't built it right the first time anyway. Martijn.
-
Hi Martijn,
Thank you for responding. I think canonical tags are the best way forward, I am looking forward to explain to the web developer that we need several hundred tags implementing!
Many thanks
-
I would definitely make sure that the pages that are expired will indeed return a 404.
In addition, what you could do as well for the duplicate content pages is use a canonical tag to point back to the original page. Wouldn't that work for you?
-
Hello,
If you want to speed up the process of deindexing, after setup all noindex tags you could upload a sitemap in GSC with all URLs you want to be crawled again.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate content w/ same URLs
I am getting high priority issues for our privacy & terms pages that have the same URL. Why would this show up as duplicate content? Thanks!
Technical SEO | | RanvirGujral0 -
Finding a specific link - Duplicating my own content
Hi Mozzers, This may be a bit of a n00b question and i feel i should know the answer but alas, here i am asking. I have a page www.website.co.uk/page/ and im getting a duplicate page report of www.website.co.uk/Page/ i know this is because somewhere on my website a link will exists using the capitalised version. I have tried everything i can think of to find it but with no luck, any little tricks? I could always rewrite the urls to lowercase, but I have downloadable software etc also on the website that i dont want to take the capitals out of. So the best solution seems to be finding the link and remove it. Most link checkers I use treat the capitalised and non capitalised as the same thing so really arent helping lol.
Technical SEO | | ATP0 -
Content Duplication and Canonical Tag settings
Hi all, I have a question regarding content duplication.My site has posted one fresh content in the article section and set canonical in the same page for avoiding content duplication._But another webmaster has taken my post and posted the same in his site with canonical as his site url. They have not given to original source as well._May I know how Google will consider these two pages. Which site will be affected with content duplication by Google and how can I solve this issue?If two sites put canonical tags in there own pages for the same content how the search engine will find the original site which posted fresh content. How can we avoid content duplication in this case?
Technical SEO | | zco_seo0 -
Duplicate Content and URL Capitalization
I have multiple URLs that SEOMoz is reporting as duplicate content. The reason is that there are characters in the URL that may, or may not, be capitalized depending on user input. A couple examples are: www.househitz.com/Pennsylvania/Houses-for-sale www.househitz.com/Pennsylvania/houses-for-sale www.househitz.com/Pennsylvania/Houses-for-rent www.househitz.com/Pennsylvania/houses-for-rent There are currently thousands of instances of this on the site. Is this something I should spend effort to try and resolve (may not be minor effort), or should I just ignore it and move on?
Technical SEO | | Jom0 -
Is it possible to be penalized as duplicate content for one keyword but not another?
I help develop an online shopping cart and after a request from management about some products not showing up in the SERP's I was able to pinpoint it down to mostly a duplicate content issue. It's a no brainer as some times new products are inserted in with copied text from the manufacturers website. I recently though stumbled across a odd problem. When we partially re-wrote the content to seem unique enough it seemed to remedy the issue for some keywords and not others. A) If you search the company name our category listing shows as #1 ahead of the manufacturers website. We always did rank for this term. B) If you search the product name our product page is listed #3 behind two other listings which belong to the manufacturer. C) If you search the keywords together as "company product" we are still being filtered out as duplicate content. When I allow the filtered results to show we are ranking #4 It's been a full month since the changes were indexed. Before I rewrite the content even further I thought I would ask to see if any one has any insight as to what could be happening.
Technical SEO | | moondog6040 -
How do I fix these duplicate URLs?
HI guys, I ran a report on my site and it shows some duplicate titles (example below). Do I need to add something to the htaccess file or another file to fix this? I understand that the search engines should only see 1 URL for the page. 2 pages have "Bikes for sale | used bikes | second hand bicycles" title pauslwebsite.com/bikes/ paulswebsite.com/bikes/index.asp Thanks
Technical SEO | | paulmund0 -
Complex duplicate content question
We run a network of three local web sites covering three places in close proximity. Each sitehas a lot of unique content (mainly news) but there is a business directory that is shared across all three sites. My plan is that the search engines only index the business in the directory that are actually located in the place the each site is focused on. i.e. Listing pages for business in Alderley Edge are only indexed on alderleyedge.com and businesses in Prestbury only get indexed on prestbury.com - but all business have a listing page on each site. What would be the most effective way to do this? I have been using rel canonical but Google does not always seem to honour this. Will using meta noindex tags where appropriate be the way to go? or would be changing the urls structure to have the place name in and using robots.txt be a better option. As an aside my current url structure is along the lines of: http://dev.alderleyedge.com/directory/listing/138/the-grill-on-the-edge Would changing this have any SEO benefit? Thanks Martin
Technical SEO | | mreeves0 -
Duplicate Content Question
Just signed up for pro and did my first diagnostic check - I came back with something like 300 duplicate content errors which suprised me because every page is unique. Turns out my pages are listed as www.sportstvjobs.com and just sportstvjobs.com does that really count as duplicate? and if so does anyone know what I should be doing differently? I thought it was just a canonical issue, but best I can tell I have the canonical in there but this still came up as a duplicate error....maybe I did canonical wrong, or its some other issue? Thanks Brian Clapp
Technical SEO | | sportstvjobs0