Www or no www

[dschreib_greenpeace.org]

right now our organization has the a robot text file set to block search engines from indexing the non-www version of our URL - example.com

looks like this

#Dynamically generated robots.txt file for example.com
User-agent: *

Disallow: /

Of course, we do allow search engines to crawl the www. version

Is this the best way to do this?

[dschreib_greenpeace.org]

Likewise to you Simon, thanks a lot for the advice!

[dschreib_greenpeace.org]

Thanks so much Kevin. Much appreciated!

[Webfor]

I wouldn't recommend setting your robots.txt to disallow indexing of the site without www. The best method is to create a domain wide 301 redirect using .htaccess. You should also set it up in Google Webmaster tools under settings and select with www as your preferred domain. Below is an example of the rewrite in .htaccess.

RewriteEngine on

RewriteCond %{HTTP_HOST} ^mysite.com [NC]

RewriteRule (.*) http://www.mysite.com/$1 [L,R=301]

Hope that helps!

Kevin

[SimonCullum]

Hi Kevin

That is a risk, you could end up blocking search engines from indexing your entire website. Usually, both www and non-www point to the same location, so one robots.txt file can apply to both (unless you've configured your site differently).

Check out  http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=44231

I'd suggest removing that Disallow command within your robots.txt file and do the following:

Within Google Webmaster Tools, go to 'Site configuration' 'Settings' ~ from there you can set how you'd like your pages to be displayed in Google's search results, either with or without the www. (or can leave Google to decide).

Regards

Simon