What is best practice to eliminate my IP addr content from showing in SERPs?

[ovenbird]

Our eCommerce platform provider has our site load balanced in a few data centers. Our site has two of our own exclusive IP addresses associated with it (one in each data center).

Problem is Google is showing our IP addresses in the SERPs with what I would assume is bad duplicate content (our own at that).

I brought this to the attention of our provider and they say they must keep the IP addresses open to allow their site monitoring software to work. Their solution was to add robots.txt files for both IP addresses with site wide/root disallows.

As a side note, we just added canonical tags so the pages indexed within the IP addresses ultimately show the correct URL (non IP address) via the canonical.

So here are my questions.

1. Is there a better way?

2. If not, is there anything else we need to do get Google to drop the several hundred thousand indexed pages at the IP address level? Or do we sit back and wait now?

[Everett]

I would allow Google to crawl those pages for a little while longer just to ensure that they see the rel canonical tags. Then once you feel that they have recrawled the IP address pages you can disallow them again if you want, thought that isn't entirely necessary if you have the rel canonical tag set up properly.

Another option would be to 301 redirect the IP version of the page to the corresponding www. version.

If they still don't drop from the index you can use the URL Removal Tool in GWT, but you will have to set up a GWT account for each of the IP domains.

[ovenbird]

Thanks. Any suggestions on how to get Google to drop these pages (make them inactive)?

[eyepaq]

Hi,

Since doing the disallow on the IP address sites, they are no longer getting crawled.

** The disavow list won't stop google crawl those domain / pages. Google will just treat those links as no follow - so they won't pass Page Rank.

You will still see  those in Web master tools, the links will still be active.

[ovenbird]

Sorry - I just thought of something that could pose a problem and was hoping to get your advice.

Since doing the disallow on the IP address sites, they are no longer getting crawled. Does that mean that the canonical tags within those IP address sites wont be able to do their work?

Or

Will the canonicals picked up from the proper domain help the search engines know they should consolidate the indexed pages from the now disallowed IP addresses?

I am seeing that the IP addresses are no longer being crawled, and the pages in their indexes about the same (not going down).

Thoughts?

[ovenbird]

Sorry - I just thought of something that could pose a problem and was hoping to get your advice.

Since doing the disallow on the IP address sites, they are no longer getting crawled. Does that mean that the canonical tags within those IP address sites wont be able to do their work?

Or

Will the canonicals picked up from the proper domain help the search engines know they should consolidate the indexed pages from the now disallowed IP addresses?

I am seeing that the IP addresses are no longer being crawled, and the pages in their indexes about the same (not going down).

Thoughts?

[ovenbird]

Thanks!

[ovenbird]

Thanks. We are getting large daily crawls (nearly 100k a day) so fingers crossed this will sort it out soon.

[eyepaq]

Hi,

The canonical solution should be enough however I would still build some xml sitemaps and submit those via Web master Tools to speed the process. You can also build some html sitemaps with a clear structure and add those in the footer - again, to speed up the proces a little bit.

If you split the content into multiple xml sitemaps you can also track the crawling process.

You should also check your crawling speed in Web Master Tools to see how many pages in avarage the google bot is hitting each day - based on those numbers you can run some prediction on how long it will take more or less for google to re crawl your pages.

If your numbers is "bad" you will need to improve it some how to help with process - it can do wonders...

Hope it helps.

[NakulGoyal]

The canonical solution you have implemented is perfect. If you have decent authority and get deep crawls every couple days, you should be fine and pages from your IP should start to disappear shortly.

I would not worry about it anymore. You are on the right track. Sit back, relax and enjoy your flight