Have I constructed my robots.txt file correctly for sitemap autodiscovery?

[Bedsite]

Hi,

Here is my sitemap:

User-agent: *

Sitemap: http://www.bedsite.co.uk/sitemaps/sitemap.xml

Directories

Disallow: /sendfriend/
Disallow: /catalog/product_compare/
Disallow: /media/catalog/product/cache/
Disallow: /checkout/
Disallow: /categories/
Disallow: /blog/index.php/
Disallow: /catalogsearch/result/index/
Disallow: /links.html

I'm using Magento and want to make sure I have constructed my robots.txt file correctly with the sitemap autodiscovery?

thanks,

[Bedsite]

Hey thanks for the response. There are about 14,000 url's in the sitemap. It shouldn't freeze up - please would you try again.

http://www.bedsite.co.uk/sitemaps/sitemap.xml

I know what you mean about the allow all

[yeagerd]

Also, here is the best place to answer your questions.

From Google:  "The Test robots.txt tool will show you if your robots.txt file is accidentally blocking Googlebot from a file or directory on your site, or if it's permitting Googlebot to crawl files that should not appear on the web. "  You can find it here

[yeagerd]

The robots.txt looks fine.  I always add an allow all, even knowing it is not necessary but it makes me feel better lol.

The problem you have is with the sitemap itself.  How big is it?  I cannot tell how many links you have because it locks up every time I go to it in both chrome and firefox.

I tried to send a tool that is designed to pull sitemaps as the SERPS do and it also freezes up.

How many links do you have?