Welcome to the Q&A Forum

hi hannah,

thx for the answer and the great insights. The problem is that for legal reason we need to show US visitors only the US version of the site. We can not give them the choice to choose other non-US versions

Hi there,

Here is our situation:we need to force an IP Redirection for our US users to www.domain.com and at the same time we have different country-specific subfolders with thei own language such as www.domain.com/fr.

Our fear is that by forcing an IP redirection for US IP, we will prevent googlebot (which has an US IP) from crawling our country-specific subfolders.

I didn't find any clear directives from Google representatives on that matter. In this video  Matt Cutts says it's always better to show Googlebot the same content as your users http://www.youtube.com/watch?v=GFf1gwr6HJw&noredirect=1, but on the other hand in that other video he says "Google basically crawls from one IP address range worldwide because (they) have one index worldwide. (They) don't build different indices, one for each country".

This seems a contradiction to me...

Thank you for your help !!

Matteo

Hello guys,

A client of ours has thousand of pages returning 404 visibile on googl webmaster tools. These are all old pages which don't exist anymore but Google keeps on detecting them. These pages belong to sections of the site which don't exist anymore. They are not linked externally and didn't provide much value even when they existed

What do u suggest us to do:

(a) do nothing

(b) redirect all these URL/folders to the homepage through a 301

(c) block these pages through the robots.txt.

Are we inappropriately using part of the crawling budget set by Search Engines by not doing anything ?

thx

Hi Keri, I am having the same issue with Mac OS

may we publicly know how to solve it ?

many thx

matteo

To follow up on this.

Look what I've found in the Google News Forum:

http://www.google.com/support/forum/p/news/thread?tid=248ef4e6fe372e91&hl=en

The problem is almost the same. Google News not indexing URLs with the trailing index.html.

The only person who answered was a Top Contributor suggesting to contact directly Google News team.

hey Roger,

Look the CNN seems to have exactly the same "problem" as we do.

http://www.google.com/#q=Obama+makes+stop+in+Los+Angeles%2C+wraps+up+campaign+swing+&fp=3986f88f9d6402d3&hl=en

They have the "/" article indexed in google news and the index.html version on the non-google news index. They did exavtly what we did, putting a rel=canonical on the "/" version to the "index.html" one. Despite this the "/" version is still the only one showing up on google news

Here is the screenshot just in case

https://skitch.com/matsutton/r5swm/obama-makes-stop-in-los-angeles-wraps-up-campaign-swing-google-search

and here the two versions of the same article:

- http://edition.cnn.com/2011/POLITICS/04/22/obama.campaign/

- http://edition.cnn.com/2011/POLITICS/04/22/obama.campaign/index.html

Hi roger and thx for the very insightful answer !

what about the fact that not a single URL ending with index.html is indexed in Google News ?

http://news.google.it/news/search?aq=f&pz=1&cf=all&ned=us&hl=en&q=inurl%3Aindex.html

compare that with the normal google index

http://www.google.it/search?q=inurl%3Aindex.html&hl=en&ned=us&tab=nw

doesn't that sound weird to you ?

matteo

Hi all,

we've been asked by a blog to help them better indexing and ranking on Google News (with the site being already included in Google News with poor results)

The blog had a chronicle URL duplication problem with each post existing with 3 different URLs:

#1)  www.domain.com/post.html (currently in noindex for editorial choices as showing all the comments)

#2) www.domain.com/post/index.html (currently indexed showing only top comments)

#3) www.domain.com/post/ (very same as #2)

We've chosen URL #2 (/index.html) as canonical URL,  and included a rel=canonical tag on URL #3 (/) linking to URL #2.
Also we've submitted  yesterday a Google News sitemap including consistently the list of URLs #2 from the last 48h . The sitemap has been properly "digested" by Google and shows that all URLs have been sent and indexed.

However if we use the site:domain.com command on Google News we see something completely different: Google News has indexed actually only some news and more specifically only the URLs #3 type (ending with the trailing slash instead of /index.html). Why ? What's wrong ?

a) Does Google News bot have problems indexing URLs ending with .index.html ? While figuring out what's wrong we've found out that http://news.google.it/news/search?aq=f&pz=1&cf=all&ned=us&hl=en&q=inurl%3Aindex.html gives no results...it seems that Google News index overall does not include any URLs ending with /index.html

b) Does Google News bot recognise  rel=canonical tag ?

c) Is it just a matter of time and then Google News will pick up the right URLs (/index.html) and/or shall we communicate Google News team any changes ?

d) Any suggestions ? OR Shall we do the other way around. meaning make URL #3 the canonical one ?

While Google News is showing these problems, Google Web search has actually well received the changes, so we don't know what to do.

Thanks for your help,

Matteo