Help with Robots.txt On a Shared Root

[Whittie]

Hi,

I posted a similar question last week asking about subdomains but a couple of complications have arisen.

Two different websites I am looking after share the same root domain which means that they will have to share the same robots.txt. Does anybody have suggestions to separate the two on the same file without complications? It's a tricky one.

Thank you in advance.

[donford]

Okay so if you have one root domain you can only have one robots.txt file.

The reason I asked for an example is in the case there was something you could put in the robots.txt to differentiate the two.

For example if you have

thisdomain.com and thatdomain.com

However if "thatdomain.com" uses a folder called shop ("thatdomain.com/shop") than you could prefix all your robots.txt file entries with /shop provided that "thisdomain.com" doesn't use the folder shop, Then all the /shop entries would only be applicable to "thatdomain.com". Does this make sense?

Don

[Whittie]

It's not so much that one is a subdomain, it's that they are as different as Google and Yahoo  yet they share the same root. I wish I could show you but I can't because of confidentiality.

The 303 wasn't put in place by me, I would have strongly suggested another method. I think it was set up so that both websites could be controlled from the same login but it's opened a can of worms for SEO.

I don't want the two separate robots files, the developer insists it has to be that way.

[donford]

Can you provide me an example of the way the domains look... Specifically where the root pages are.

Domain1.com

forums.domain1.com ?

Additionally, if you are redirecting 303 one of the domains to the other why do you want two different robots.txt files? The one being 303 will always redirect to the other...?

Depending on the structures you can create one robots.txt file that deals with 2 different domains provided there is something unique about the root folders.

[Whittie]

Thanks for your help so far.

The two different websites are different name domains but share the same root as it's been built this way on Typo3. I don't know of the developer's justification for the 303, it's something I wish we could change.

I'm not sure if there are specific tags you can put in the sole robots.txt to differentiate the two, have read a few conflicting arguments about how to do it.

[donford]

Okay so if you're using a 303 then you're saying the content you want for X site is actually located at Y site.Which means you do not have 2 different sub domains. So there is no need for 2 robots.txt files and your developer is correct you can't use 2 robots.txt files. Since one site would be pointing to the other you only have one sub-domain.

However, 303 is in general a poor way to use a redirect and likely should be 301.. but I would have to understand why the 303 is being used to say that with 100% certainty. See a quick article about 303 here..

Hope this answers the question,

Don

[Whittie]

It's Fasthosts. The developer is certain that we can't use the two separate robots files. The second website has been set up on a 303.

[donford]

What host are you using?

[Whittie]

The developer of the website insists that they have to share the same robots.txt, I am really not sure how he's set it up this way. I am beyond befuddled with this!

[donford]

The subdomain has to be separated from the root in some fashion. I would assume depending on your host that there is a separate folder for the subdomain stuff. Otherwise it would be chaos. Say you installed forums on your forum subdomain and a e-commerce on your shop subdomain... which index.php page would be served?

There has to be some separation, review your file manager and look for the sub-domain folders. Once found you simply put a robots.txt into each of those folders.

Hope this helps,

Don