Load balanced Site
-
Our client ecommerce site load from 3 different servers using load balancing.
abc.com : IP: 222.222.222
Abc.com: IP: 111.111.111
For testing purpose 111.111.111 also point to beta.abc.com
Now google crawling site beta.abc.com
If we block beta.abc.com using robots.txt it will block google bot also , since beta.abc.com is really abc.com
I know its confusing but I been trying to figure out. Ofcourse I can ask my dev to remove beta.abc.com make a seperate code and block it using .htaccess
-
Our solution for this is to use http authentication. So our dev sites require a simple password to access.
Here is an example: http://dev.zeta-commerce.com/
This keeps the bots out and avoids the risk of a robots.txt file being released accidentally to the live site.
-
Maybe I'm not understanding, but if you can differentiate on the server the difference between beta.abc.com and abc.com, subdomains can have different robots.txt files, so you should be able to serve a file for http://beta.abc.com/robots.txt that disallows everything, with
User-agent: * Disallow: /
and a different robots.txt file for http://abc.com/robots.txt. If you do so, it shouldn't block Googlebots access to abc.com.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to avoid adult traffic to site?
A client of ours is increasingly getting a lot of adult traffic to their site, where they show up only for adult searches and not at all for relevant searches. How can we stop Google associating their site with adult content? Here's a blog example, giving advice to parents on girls and body image issues: https://www.commonsensemedia.org/blog/girls-and-body-image keywords driving traffic to this page are all around images for 'young nude girls' etc.
Intermediate & Advanced SEO | | MediaCause0 -
Linking to one of my own sites, from my site
Hi experts, I own a site for castingjobs (Site1) and a site for selling paintings (Site2). In a long time, I've had a link at the bottom of Site1, linking to Site 2. (Basicaly: Partnerlink: Link site 2). Site1 is for me the the only important site, since it's where Im making my monthly revenue. I added the link like 5 years ago or so, to try to boost site 2. My question is:
Intermediate & Advanced SEO | | KasperGJ
1. Is it somehow bad for SEO for site 1, since the two sites have nothing to do with each other, they are basically just owned by me.
2. Would it make sense to link from Site 2 to Site 1 indstead?0 -
New site, new URL, lots of custom content. Load it all or "trickle" it over time?
New site, new URL, lots of custom content. Load it all or "trickle" it over time? Would it make a difference in terms of ranking the site? Interested in your thoughts. Thanks! BBuck!
Intermediate & Advanced SEO | | BBuck0 -
How are these sites ranking!?!
One of our clients is in the insurance industry and over the last 12 months we have seen an increasing number of low quality, newly registered, spammy sites achieving top 5 rankings for major keywords, which in turn is having an adverse effect on the rankings for our client. Does anyone have any idea how the following sites have managed to do this: http://www.multiquotetaxi.co.uk/ - 2nd for taxi insurance http://www.motortradefast.co.uk/ - 1st for motor trade insurance http://www.traders-insurance.com/ - 3rd for motor trade insurance http://www.multiquotefleet.co.uk/ - 1st for fleet insurance We have tried reporting the above sites, tried holding out to see if they get penalised and tried figuring out what they have done ourselves but cannot see how they have managed it. Any ideas at all?
Intermediate & Advanced SEO | | instinctive0 -
Why are these sites outranking me?
I am trying to rank for the phrase "a link between worlds walkthrough" I am on page 1 but there are several results that just outranks me and I cannot see any reason that they would be doing so. My site is hiddentriforce.com/a-link-between-worlds/walkthrough/ For that page I have 5 linking domains, varied anchor text that spans from things like "here" to a variety of related phrases. All of the links come from really good sites My page has 1400 likes, 90 shares, and about 20 each in tweets and +'s DA of 44 PA of 37 The 4 and 5 ranked sites both have WAY less social interactions, lower PA and DA, less links, etc Yet they outrank me why?
Intermediate & Advanced SEO | | Atomicx0 -
How come this site does so well?
Hi Guys, It's bugging the crap out of me why this site does so well http://www.stagedinburgh.com/ when I look at it's link profile its so weak and terrible plus many links comes from the sites they own. Somehow the site out ranks many sites for search terms like edinburgh stag party, edinburgh stag do, edinburgh stag weekends. Am I missing something? They seem to only have links from 13 domains and they aint great. What am I missing?
Intermediate & Advanced SEO | | PottyScotty0 -
How to remove an entire site from Google?
Hi people, I have a site with around 2.000 urls indexed in google, and 10 subdomains indexed too, which I want to remove entirely, to set up a new web. Which is the best way to do it? Regards!
Intermediate & Advanced SEO | | SeoExpertos0 -
Load balancing - duplicate content?
Our site switches between www1 and www2 depending on the server load, so (the way I understand it at least) we have two versions of the site. My question is whether the search engines will consider this as duplicate content, and if so, what sort of impact can this have on our SEO efforts? I don't think we've been penalised, (we're still ranking) but our rankings probably aren't as strong as they should be. The SERPs show a mixture of www1 and www2 content when I do a branded search. Also, when I try to use any SEO tools that involve a site crawl I usually encounter problems. Any help is much appreciated!
Intermediate & Advanced SEO | | ChrisHillfd0