Good robots txt for magento

[rijwielcashencarry040]

Dear Communtiy,

I am trying to improve the SEO ratings for my website www.rijwielcashencarry.nl (magento). My next step will be implementing robots txt to exclude some crawling pages.
Does anybody have a good magento robots txt for me? And what need i copy exactly?

Thanks everybody!

Greetings,

Bob

[RuthBurrReedy]

This is fine, as long as you don't want to exclude robots from crawling any part of your site.

[Fahad5Saleem]

Me to have this problem, if someone can help with setting root.txt  
 my web

current configuration is

Sitemap: http://www.myweb/sitemap.xml
User-agent: *
Disallow:

THIS IS GOOD ?

[rijwielcashencarry040]

Hi Ruth,

Also thanks for your response!

Greetings,

Bob

[rijwielcashencarry040]

Hi Peter,

Thanks for your response! I am going to follow up your advice and build a good Robots TXT.

Greetings,

Bob

[RuthBurrReedy]

Peter is correct - your search, admin and user pages are common pages to block for Magento. What you block is up to you, though. Don't forget that a page that is blocked by robots.txt can still be found by search engines, so if it's a page that will contain private information you should protect it with a password.

[MattRoney]

Hi there! Did Peter's response take care of this for you? If so, please mark it as a "Good Answer."

[rootwaysinc]

Hi,

Creating robots.txt file for the site is one of the most important thing, you need to understand your website or stores basic needs what to keep private and what to make public; I think you need to block some part in your magento site like your search pages (?*sid) and admin pages, user dashboard pages, here is an example links Robots.txt for Magento and Robots.txt File Examples