One server, two domains - robots.txt allow for one domain but not other?
-
Hello,
I would like to create a single server with two domains pointing to it. Ex: domain1.com -> myserver.com/ domain2.com -> myserver.com/subfolder. The goal is to create two separate sites on one server.
I would like the second domain ( /subfolder) to be fully indexed / SEO friendly and have the robots txt file allow search bots to crawl. However, the first domain (server root) I would like to keep non-indexed, and the robots.txt file disallowing any bots / indexing.
Does anyone have any suggestions for the best way to tackle this one?
Thanks!
-
I am assuming that you are using Cpanel on the host for the site. Just add the second domain as an addon domain. Because of the way that things are set up in Cpanel and the way it handles redirects, the crawlers will not realize the local directory structure that one site is sitting in the sub folder of another site. They will be viewed as if they are two different sites on different servers. So you would handle the robtos.txt like they were on two different servers and just disregard anything about one being in the sub folder of another.
-
A Records
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Domain change recommendations
We recently migrated one of our websites to a new domain. Obviously we were expecting a decrease in traffic initially, but it has actually gone down by 70% week-over-week since we made the switch. We set up a 301 redirect from the old domain to the new domain, changed all internal links to the new domain and changed all inbound links that we owned to the new domain. Our research suggested the best way to approach a domain change was by keeping it simple and not making too many changes at once. So my questions are: 1. Are these the kinds of results we should expect initially after a domain change? And if not, 2. What are the steps we should take from here? Thanks!
Technical SEO | | gouldtr0 -
Which domain we should continue with?
Hello All, We are working with a client who had manual penalty from Google. We worked on that and now penalty has been removed. Client had already started working on the new domain and now the big dilemma is- Which domain should we continue with? Old or New? We are suggesting them to continue with the old one as that domain had good PR, good backlinks, better visibility on their social profiles etc. What do you suggest? any inputs are highly appreciated. Thanks
Technical SEO | | sachin-sv0 -
Two META Robots tags on a page - which will win?
Hi, Does anybody know which meta-robots tag will "win" if there is more than one on a page? The situation:
Technical SEO | | jmueller
our CMS is not very flexible and so we have segments of META-Tags on the page that originate from templates.
Now any author can add any meta-tag from within his article-editor.
The logic delivering the pages does not care if there might be more than one meta-robots tag present (one from template, one from within the article). Now we could end up with something like this: Which one will be regarded by google & co?
First?
Last?
None? Thanks a lot,
Jan0 -
What is cross domain?
what is cross domain? can any one explain in simple language ?
Technical SEO | | constructionhelpline0 -
Should we 301 redirect our old domain to the new domain
We have a product that when started was under the domain appnowgo.com. We've since changed the name and the domain is now knackhq.com. The latter domain doesn't rank nearly as well as the former for many of the keywords we are targeting. For example... "online database builder" and "web app builder" are two of those keywords. Obviously having app in the domain is not a bad thing but it is our old name. The question is, should we 301 the appnowgo.com domain to knackhq.com? Or should we use that better rank and just link users to knackhq.com from the appnowgo.com site until we can increase our ranking for knackhq.com? We don't plan to update the content on appnowgo.com anymore and we obviously don't want to drop off rank if at all possible. Thanks! Eric
Technical SEO | | sitestrux0 -
¿Is a good idea to have in the same hosting (disck space) two diferent domains?
Hello. I'm talking about Google Spain. I have two domains allocated in the same site. One, including the keyword selected for the domain The other is the branding name. In this way, my client can print the branding name in their cards. But for us is easier to get rankings and (I think) quickly also. In this way, what I hope, is that if someone is looking for the service in Google, the domain with the keyword appears. But the site can also be accesed by the branding domain (which is shorter and more friendly.
Technical SEO | | OkTuWeb0 -
Grabbing Expired Domains
How hard is it to grab expired domains? I have my eye on a domain that is expiring in 3 days, but I don't think it's quite that simple. Doesn't it go through months of waiting to become available? Is there an easy way to grab domains that are set to expire? Are the services that offer this type of service good? And who do you guys recommend?
Technical SEO | | applesofgold0 -
Should I set up a disallow in the robots.txt for catalog search results?
When the crawl diagnostics came back for my site its showing around 3,000 pages of duplicate content. Almost all of them are of the catalog search results page. I also did a site search on Google and they have most of the results pages in their index too. I think I should just disallow the bots in the /catalogsearch/ sub folder, but I'm not sure if this will have any negative effect?
Technical SEO | | JordanJudson0