Old pages still in index

[ssiebn7]

Hi Guys,

I've been working on a E-commerce site for a while now. Let me sum it up :

• February new site is launched
• Due to lack of resources we started 301's of old url's in March
• Added rel=canonical end of May because of huge index numbers (developers forgot!!)
• Added noindex and robots.txt on at least 1000 urls.
• Index numbers went down from 105.000 tot 55.000 for now, see screenshot (actual number in sitemap is 13.000)

Now when i do site:domain.com there are still old url's in the index while there is a 301 on the url since March!

I know this can take a while but I wonder how I can speed this up or am doing something wrong. Hope anyone can help because I simply don't know how the old url's can still be in the index.

4cArHPH.png

[ssiebn7]

Hi Dan,

Thanks for the answer!

Indexation is already back to 42.000 so slowly going back to normal

And thanks for the last tip, that's totally right. I just discovered that several pages had duplicate url's generated so by continually monitoring we'll fix it !

[evolvingSEO]

Hi There

To noindex pages there are a few methods;

• use a meta noindex without robots.txt - I think that is why some may not be removed. The robots.txt block crawling so they can not see the noindex.

• use a 301 redirect - this will eventually kill off the old pages, but it can definitely take a while.

• canonical it to another page. and as Chris says, don't block the page or add extra directives. If you canonical the page (correctly), I find it usually drops out of the index fairly quickly after being crawled.

• use the URL removal tool in webmaster tools + robots.txt or 404. So if you 404 a page or block it with robots.txt you can then go into webmaster tools and do a URL removal. This is NOT recommended though in most normal cases, as Google prefers this be for "emergencies".

The only method that removes pages within a day or two guaranteed is the URL removal tool.

I would also examine your site since it is new, for something that is causing additional pages to be generated and indexed. I see this a lot with ecommerce sites where they have lots of pagination, facets, sorting, etc and those can generate lots of other pages which get indexed.

Again, as Chris says, you want to be careful to not mix signals. Hope this all helps!

-Dan

[ssiebn7]

Hi Chris,

Thanks for your answer.

I'm either using a 301 or noindex, not both of course.

Still have to check the server logs, thanks for that!

Another weird thing. While the old url is still in the index, when i check the cache date it's a week old. That's what i don't get. Cache date is a week old but Google still has the old url in the index.

[Chris.Menke]

It can take months for pages to fall out of Google's index have you looked at your log files to verify that googlebot is crawling those pages?.  Things to keep in mind:

• If you 301 a page, the rel=canonical on that page will not be seen by the bot (no biggie in your case)
• If you 301 a page, a meta noindex will not be seen by the bot
• It is suggested not to use the robots.txt to no index a page  that is being 301 redirected--as the redirect may not be seen by Google.