Need Help With Robots.txt on Magento eCommerce Site

[JerDoggMckoy]

Hello, I am having difficulty getting my robots.txt file to be configured properly. I am getting error emails from Google products stating they can't view our products because they are being blocked, and this past week, in my SEO dashboard, the URL's receiving search traffic dropped by almost 40%.

Is there anyone that can offer assistance on a good template robots.txt file I can use for a Magento eCommerce website?

The one I am currently using was found at this site here: e-commercewebdesign.co.uk/blog/magento-seo/magento-robots-txt-seo.php - However, I am getting problems from Google now because of it.

I searched and found this thread here: http://www.magentocommerce.com/wiki/multi-store_set_up/multiple_website_setup_with_different_document_roots#the_root_folder_robots.txt_file - But I felt like maybe I should get some additional help on properly configuring a robots for a Magento site.

Thanks in advance for any help. Please, let me know if you need more info to provide assistance.

[Francisco_Meza]

You better back up your DB before doing that. Anyway, take a look at this MagentoConnect extension http://www.magentocommerce.com/magento-connect/MageWorx.com/extension/2852/seo-suite-enterprise#overview

or this one (it's by the same company

http://www.mageworx.com/seo-suite-pro-magento-extension.html

[JerDoggMckoy]

Thank you very much. We'll give that a shot and see how it goes. What started us tinkering with the robots file in the first place is that Bing Shopping told us it couldn't crawl our product images. Plus, our pdf files for product specs and manuals are all listed within the media folder. Do you have a suggestion for this? I would think we would get rid of "Disallow: /media/" and replace it with the following (what do you think?):

Disallow: /media/aitmanufacturers/
Disallow: /media/bigtom_media/
Disallow: /media/css/
Disallow: /media/downloadable/
Disallow: /media/easybanner/
Disallow: /media/geoip/
Disallow: /media/icons/
Disallow: /media/import/
Disallow: /media/js/
Disallow: /media/productsfeed/
Disallow: /media/sales/
Disallow: /media/tmp/
Disallow: /media/UPS/

[tylerfraser]

Hello,

Below is what I use. You need to have the modrewrite enabled if you are going to disallow index.php and even then it's still very risky. This may be part of the issue. Robots.txt is so important, but you need to know what you are doing. Especially when disallowing as much as that UK site is.

Tyler

User-agent: *

Disallow: /*?

Disallow: /*.js$

Disallow: /*.css$

Disallow: /checkout/

Disallow: /catalogsearch/

Disallow: /review/

Disallow: /app/

Disallow: /downloader/

Disallow: /images/

Disallow: /js/

Disallow: /lib/

Disallow: /media/

Disallow: /*.php$

Disallow: /pkginfo/

Disallow: /report/

Disallow: /skin/

Disallow: /var/

Disallow: /customer/

Disallow: /enable-cookies/

Sitemap: http://domain.com/sitemap.xml