Www or no www
-
right now our organization has the a robot text file set to block search engines from indexing the non-www version of our URL - example.com
looks like this
#Dynamically generated robots.txt file for example.com
User-agent: *Disallow: /
Of course, we do allow search engines to crawl the www. version
Is this the best way to do this?
-
Likewise to you Simon, thanks a lot for the advice!
-
Thanks so much Kevin. Much appreciated!
-
I wouldn't recommend setting your robots.txt to disallow indexing of the site without www. The best method is to create a domain wide 301 redirect using .htaccess. You should also set it up in Google Webmaster tools under settings and select with www as your preferred domain. Below is an example of the rewrite in .htaccess.
RewriteEngine on
RewriteCond %{HTTP_HOST} ^mysite.com [NC]
RewriteRule (.*) http://www.mysite.com/$1 [L,R=301]
Hope that helps!
Kevin
-
Hi Kevin
That is a risk, you could end up blocking search engines from indexing your entire website. Usually, both www and non-www point to the same location, so one robots.txt file can apply to both (unless you've configured your site differently).
Check out http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=44231
I'd suggest removing that Disallow command within your robots.txt file and do the following:
Within Google Webmaster Tools, go to 'Site configuration' 'Settings' ~ from there you can set how you'd like your pages to be displayed in Google's search results, either with or without the www. (or can leave Google to decide).
Regards
Simon
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
We switched the domain from www.blog.domain.com to domain.com/blog.
We switched the domain from www.blog.domain.com to domain.com/blog. This was done with the purpose of gaining backlinks to our main website as well along with to our blog. This set us very low in organic traffic and not to mention, lost the backlinks. For anything, they are being redirected to 301 code. Kindly suggest changes to bring back all the traffic.
Technical SEO | | arun.negi0 -
I have made my new website live. But while checking in Google it is not showing in search result ( site: www.oomfr.com ). Can anybody please advice.
Hi Team, I have made my new website live. But while checking in Google it is not showing in search result ( site: www.oomfr.com ). Can anybody please advice.
Technical SEO | | nlogix0 -
Duplicate Page Content for www and non-www. Help!
Hi guys, having a bit of a tough time here... MOZ is reporting duplicate content for 21 pages on eagleplumbing.co.nz, however the reported duplicate is the www version of the page. For example: http://eagleplumbing.co.nz and http://www.eagleplumbing.co.nz are considered duplicates (see screenshot attached) Currently in search console I have just updated the non-www version to be set as the preferred version (I changed this back and forth twice today because I am confused!!!). Does anyone know what the correct course of action should be in this case? Things I have considered doing include: changing the preferred version to the www version in webmaster tools, setting up 301 redirects using a wordpress plugin called Eggplant 301 redirects. I have been doing some really awesome content creation and have created some good quality citations, so I think this is only thing that is eaffecting my rank. Any help would be greatly appreciated. view?usp=sharing
Technical SEO | | QRate0 -
What is the best practice to seperate different locations and languages in an URL? At the moment the URL is www.abc.com/ch/de. Is there a better way to structure the URL from an SEO perspective?
I am looking for a solution for using a new URL structure without using www.abc.com**/ch/de** in the URL to deliver the right languages in specific countries where more than one language are spoken commonly. I am looking forward to your ideas!
Technical SEO | | eviom0 -
Url canonicalization: www. to http://
Hey there. Sorry for the simple question but I recently redesigned a site and published with WordPress, in the process the domain structure changed from being www. to http:// . My question is does this change affect the value we get from links pointing to the old www. domain structure? The reason I ask is that the old site had a domain authority of 36 with OSE and a couple of hundred links but the new site address shows as having zero domain authority and zero links. Is there some best practise I should be following to retain link value?
Technical SEO | | Luia0 -
Setting preferred domain as www or none www
Way back before panda I used to rank for certain keywords pretty well. Of course like many others after panda I lost some of those rankings. I have been getting better since then so its not that bad. I was poking around in Google Webmaster Tools and I noticed something which I need some clarification in. History my site freescrabbledictionary.com used to be indexed as a none www. Then some time ago I can't remember when I set it to www. Tonight I was looking through my webmaster tools and I noticed something that did not make sense to me. In my content keywords section for the none www my list is as follows Content Keywords <form action="https://www.google.com/webmasters/tools/keywords-list?hl=en&siteUrl=http://freescrabbledictionary.com/" method="GET"> Keyword Significance 1. scrabble 2. words (2 variants) 3. dictionary 4. cheat 5. finder 6. friends 7. maker (2 variants) 8. noun 9. letter (2 variants) 10. hasbro 11. mattel 12. spear 13. found (2 variants) 14. sowpods 15. freescrabbledictionary 16. builder 17. affiliated 18. search 19. solver 20. lists </form> Then I looked at my www lists and its Content Keywords <form action="https://www.google.com/webmasters/tools/keywords-list?hl=en&siteUrl=http://www.freescrabbledictionary.com/" method="GET"> Keyword Significance 1. words (3 variants) 2. scrabble (2 variants) 3. letter (4 variants) 4. points 5. cheat (3 variants) 6. friends (2 variants) 7. finder (2 variants) 8. anagram (2 variants) 9. dictionary 10. tool (2 variants) 11. hasbro 12. mattel 13. spear 14. game (4 variants) 15. mobile 16. affiliated (3 variants) 17. berkshire 18. canada 19. calculations (5 variants) 20. coming (4 variants) </form> My none www version has the order (especially the first 5 keywords) that I want, my www version is no were near it. If I change back to the none www version could I possible see an change in rank? or can it effect it if I change it? I am starting to think I shot myself in the foot when I switched...
Technical SEO | | cbielich0 -
Site:www.tld.com rank is it a measure of googles per page importance?
Hello, does the order of pages in a site:www.tld.com search show how important each page is to google? what if the homepage is not the first result?
Technical SEO | | adamzski0 -
Should I 301 my non-www accesses to www accesses?
We have external links pointing to both mydomain.com and www.mydomain.com. I read this: http://www.stepforth.com/resources/web-marketing-knowledgebase/non-www-redirect/ and wondered if I should add this to my .htaccess file: RewriteCond %{HTTP_HOST} ^mydomain.com
Technical SEO | | scanlin
RewriteRule (.*) http://www.mydomain.com/$1 [R=301,L] so that the link juice all flows to the www version of the site? Any reason not to do it?0