Welcome to the Q&A Forum

Hi all,

sorry for the delay, I am away on a business trip, this is why I stopped communicating the past few days.

I can confirm that the latest entries (those after March) come as a single instance.
However there are some minor exceptions like the one here

Example of a recent article indexed in both desktop (even though desktop url is not the canonical) and mobile URL
https://www.google.gr/search?q=site:neakriti.gr&biw=1527&bih=899&source=lnms&sa=X&ved=0ahUKEwiIxODGt5_MAhUsKpoKHdcUAkYQ_AUIBigA&dpr=1.1#q=site:neakriti.gr+1315539&tbs=qdr:w&filter=0

Also I noticed that with the "alternate" and "canonical" links the mobile version of the site doesn't get indexed anymore (with minor exceptions like the one above).

Thank you Kristina,

I know about the URL structure, I have been trying the past few months to cleanse this site that I was not involved in its creation.  It has several more SEO problems that have either been fixed or not yet, but we are talking about more than 50 SEO problems I've found so far - most of these critical.

On the sitemap that I built, the junk pages do not exist, and because this is sitemap I have written myself, I can easily make another containing the articles that I have removed (just reverse a part of my select query for the sitemap to get the ones I have removed).

http://www.neakriti.gr/webservices/sitemap-index.aspx

So far I implemented the last of your suggestions and here is an example:

This is a valid article page
http://www.neakriti.gr/?page=newsdetail&DocID=1314221 - (Status Code: 200)

This is a non existent article page (never existed at the first place) - (Status Code: 404)
http://www.neakriti.gr/?page=newsdetail&DocID=12345678

This is one of the articles that I removed from sitemap and site - (Status Code: 410)
http://www.neakriti.gr/?page=newsdetail&DocID=894052

Also I would like you to take a look at another question about the same site and see that it can relate to this question with garbage articles too...
https://moz.com/community/q/multiple-instances-of-the-same-article

Thank you so much!

Thank you for your response, I will take a look at this.

However I have two questions regarding your suggestion

1. Since I have canonical links at the loading page, doesn't that resolve the issue?
2. the printerfriendly variation has a noindex meta at the head, shouldn't that be taken into account?
3. Can I put regular expressions in my robots.txt?  How can I block url params?  Because printerfriendly and newsdetailsports are values of the "page" GET param

Infact the printerfriendly contains canonical link and noindex meta to inform search engines not to index content, and let them know where the original content exists

Hi, I'm having a problem I cannot solve about duplicate article postings.

As you will see from the attached images, I have a page with multiple variants of the same URL in google index and as well as duplicate title tag in the search console of webmasters tools. Its been several months I have been using canonical meta tags to resolve the issue, aka declare all variants to point to a single URL, however the problem remains. Its not just old articles that stay like that, even new articles show the same behaviour right when they are published even thought they are presented correctly with canonical links and sitemap as you will see from the example bellow.

Example URLs of the attached Image

1. All URLs belonging to the same article ID, have the same canonical link inside the html head.

2. Also because I have a separate mobile site, I also include in every desktop URL an "alternate" link to the mobile site.

3. At the Mobile Version of the Site, I have another canonical link, pointing back to the original Desktop URL. So the mobile site article version also has

4. Now, when it comes to the xml sitemap, I pass only the canonical URL and none of the other possible variants (to avoid multiple indexing), and I also point to the mobile version of the article.
   <url><loc>http://www.neakriti.gr/?page=newsdetail&DocID=1300357</loc>
     <xhtml:link rel="alternate" media="only screen and (max-width: 640px)" href="http://mobile.neakriti.gr/fullarticle.php?docid=1300357"><lastmod>2016-02-20T21:44:05Z</lastmod>
     <priority>0.6</priority>
     <changefreq>monthly</changefreq>
     image:imageimage:lochttp://www.neakriti.gr/NewsASSET/neakriti-news-image.aspx?Doc=1300297</image:loc>
       image:titleΟΦΗ</image:title></image:image></xhtml:link></url>

The above Sitemap snippet Source: http://www.neakriti.gr/WebServices/sitemap.aspx?&year=2016&month=2
The main sitemap of the website: http://www.neakriti.gr/WebServices/sitemap-index.aspx

Despite my efforts you see that webmasters tools reports three variants for the desktop URL, and google search reports 4 URLs (3 different desktop variant urls and the mobile url).

I get this when I type the article code to see if what is indexed in google search: site:neakriti.gr 1300297

So far I believe I have done all I could in order to resolve the issue by addressing canonical links and alternate links, as well as correct sitemap.xml entry.  I don't know what else to do... This was done several months ago and there is absolutelly no improvement.

Here is a more recent example of an article added 5 days ago (10-April-2016), just type
site:neakriti.gr 1300357
at google search and you will see the variants of the same article in google cache.  Open the google cached page, and you will see the cached pages contain canonical link, but google doesn't obey the direction given there.

Please help!

duplicate-articles.jpg duplicate-articles-in-index.jpg

Thank you,

so you suggest that based on my date based query, instead of blocking everything before that date blindly, keep blocking it with 410, while anything that doesn't exist anyway return 404.

Also another question, about the blocked articles that return 410, should I put their URLs back on the xml sitemap or not?

Thank you for your response.

I defenintelly cannot use noindex because as I explained I changed all articles prior to the minimum given date to return 404.  So this content is not visibly available on the web in order to contain a noindex directive.  Unless you mean to have it at my custom 404 page, where yes its there.

Also there is no folder to associate in robots, since they are in ugly form of GET params like DOCID=12345.  So given that, there are thousands of DocIDs that are junk and removed, and thousands that are the actuall articles.

So I assumed that creating a "deleted articles" sitemap where each <url>will contain an <expires>2016-06-01</expires> tag seemed the most logical thing, but I am afraid its for "custom search engines", rather than for normal de-index requests as its provided bellow</url>

https://developers.google.com/custom-search/docs/indexing#on-demand-indexing

Hi,

I am trying to cleanse a news website.  When this website was first made, the people that set it up copied all kinds of articles they had as a newspaper, including tests, internal communication, and drafts.  This site has lots of junk, but this kind of junk was on the initial backup, aka before 1st-June-2012.  So, removing all mixed content prior to that date, we can have pure articles starting June 1st, 2012!

Therefore

1. My dynamic sitemap now contains only articles with release date between 1st-June-2012 and now
2. Any article that has release date prior to 1st-June-2012 returns a custom 404 page with "noindex" metatag, instead of the actual content of the article.

The question is how I can remove from the google index all this junk as fast as possible that is not on the site anymore, but still appears in google results?

I know that for individual URLs I need to request removal from this link
https://www.google.com/webmasters/tools/removals

The problem is doing this in bulk, as there are tens of thousands of URLs I want to remove.  Should I put the articles back to the sitemap so the search engines crawl the sitemap and see all the 404?  I believe this is very wrong.  As far as I know this will cause problems because search engines will try to access non existent content that is declared as existent by the sitemap, and return errors on the webmasters tools.

Should I submit a DELETED ITEMS SITEMAP using the <expires>tag? I think this is for custom search engines only, and not for the generic google search engine.
https://developers.google.com/custom-search/docs/indexing#on-demand-indexing</expires>

The site unfortunatelly doesn't use any kind of "folder" hierarchy in its URLs, but instead the ugly GET params, and a kind of folder based pattern is impossible since all articles (removed junk and actual articles) are of the form:
http://www.example.com/docid=123456

So, how can I bulk remove from the google index all the junk... relatively fast?