Robots.txt and Magento

[EcomLkwd]

HI,

I am working on getting my robots.txt up and running and I'm having lots of problems with the robots.txt my developers generated. www.plasticplace.com/robots.txt

I ran the robots.txt through a syntax checking tool (http://www.sxw.org.uk/computing/robots/check.html) This is what the tool came back with:  http://www.dcs.ed.ac.uk/cgi/sxw/parserobots.pl?site=plasticplace.com  There seems to be many errors on the file.

Additionally, I looked at our robots.txt in the WMT and they said the crawl was postponed because the robots.txt is inaccessible. What does that mean?

A few questions:

1. Is there a need for all the lines of code that have the “#” before it? I don’t think it’s necessary but correct me if I'm wrong.

2. Furthermore, why are we blocking so many things on our website? The robots can’t get past anything that requires a password to access anyhow but again correct me if I'm wrong.

3.  Is there a reason Why can't it just look like this:

User-agent: *

Disallow: /onepagecheckout/

Disallow: /checkout/cart/

I do understand that Magento has certain folders that you don't want crawled, but is this necessary and why are there so many errors?

[Felip3]

Yes your short robots.txt idea would create a huge problem.

In your Magento admin if you click in the menu Catalog > URL Rewrite Management

You will see the magento feature  that creates all the "pretty urls", in that page you will see a table. If get value from  Target path column and copy and paste after your site domain, for example domain.com/value_in_target_path...

You'll see that the page loads fine, you don't want Google to rank those pages with the "messy" URL so that's why you need all those stuff in your robots.txt

[EcomLkwd]

I am bit confused. Are you saying that technically my Magento site has two different urls that can both be indexed; one with a (messy) url and another with a vanity url?  This would create major duplicate content issues! The robots.txt would not solve such a complex issue.

Am I missing something?

[EcomLkwd]

My developer said they custom configured it to block the files they needed according to Magento.

You think I can simply make it look like this:

User-agent: *

Disallow: /onepagecheckout/

Disallow: /checkout/cart/

and then disable it in WMT?

[Felip3]

3.  Is there a reason Why can't it just look like this:

Yes, It would generate a lot of duplicates issues, for example your  robots.txt you have the follow line:

Disallow: /catalog/category/view/ -> That's the "real" category URL, you can access any category on magento by /catalog/category/view/id or by the "pretty" URL.
Because you disallow the "real: URL only the pretty URL will be viable for search engines. 

This same rule apply for many other parts of the robots.txt.

[James77]

I assume this is a robots.txt that has been automatically created by Magento? - or has it been created by a developer?

I ran it through a tool and it showed 1 error and 10 warnings - so i would say you definitely need to do something about it.

The reason for all those disallows is to try and stop search engine indexing them (whether they would even find them to index them if they were not there is debatable).

What you could do is set up robots.txt as you have suggested and then stop the SE's indexing the directories or pages you don't want in appropriate webmaster tools.

I don't like displaying a lot of 'don't index' paths in the robots texts as it is pretty much telling any hacker or nasty spider where your weak points may be.