Load balanced Site
-
Our client ecommerce site load from 3 different servers using load balancing.
abc.com : IP: 222.222.222
Abc.com: IP: 111.111.111
For testing purpose 111.111.111 also point to beta.abc.com
Now google crawling site beta.abc.com
If we block beta.abc.com using robots.txt it will block google bot also , since beta.abc.com is really abc.com
I know its confusing but I been trying to figure out. Ofcourse I can ask my dev to remove beta.abc.com make a seperate code and block it using .htaccess
-
Our solution for this is to use http authentication. So our dev sites require a simple password to access.
Here is an example: http://dev.zeta-commerce.com/
This keeps the bots out and avoids the risk of a robots.txt file being released accidentally to the live site.
-
Maybe I'm not understanding, but if you can differentiate on the server the difference between beta.abc.com and abc.com, subdomains can have different robots.txt files, so you should be able to serve a file for http://beta.abc.com/robots.txt that disallows everything, with
User-agent: * Disallow: /and a different robots.txt file for http://abc.com/robots.txt. If you do so, it shouldn't block Googlebots access to abc.com.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Onpage optimising for multiple sites
I’ve been given the task of optimizing a company’s websites (15 in total) that has multiple websites selling the same product. In terms of optimizing them, can I use the same set of meta descriptions, page title tags and key words for them all or do I need to produce a different set for each? The sites are for independently branded companies that are set up in a franchise-like arrangement. They all exclusively sell the parent companies joinery products
Intermediate & Advanced SEO | | aplnzoctober180 -
Domain Change Before or After Site Revamp?
In the last year traffic to our site has dropped in half and ranking has dropped significantly. Very little no content has been added in that time. We would now like to improve ranking by adding new content. 2 domains effectively exist for the site. The existing domain is www.nyc-officespace-leader.com. But www.metro-manhattan.com redirects to www.nyc-officespace-leader.com. Our company is Metro Manhattan Office Space, Inc.. We registered www.metro-manhattan.com and created the redirect to www.nyc-officespace-leader.com in 2012. www.nyc-officespace-leader.com was registered in 2006. Many links to the site show www.metro-manhattan.com and I believe this may be a source of confusion for Google. Would it be best to make the domain consistent at this time by redirecting it once and for all and to do so before adding new content? If this is done correctly can we avoid taking a hit on ranking? Note: -www.nyc-officespace-leader.com is the old domain.
Intermediate & Advanced SEO | | Kingalan1
-www.metro-manhattan is the new domain but has existed since 2012 and has been redirecting to the old domain since then
-The company name is Metro Manhattan Office Space (similar in branding to the new domain) Am I correct in assuming that having the 2 domains may be causing issues with Google involving domain authority? Change the domain before adding content or add content before?0 -
Are Their Any SEO Dangers When Cleaning Up a Site
I'm doing some housekeeping on my website. Removing old blogs that are out of date (2008) or things have moved on. The blogs I'm removing are being 301'd to relevant newer blogs. Can this type of clean up cause any problems that affect the optimisation of a site? Looking forward to hearing your views. Christina
Intermediate & Advanced SEO | | ChristinaRadisic0 -
Tips for Link Building for Mobile Sites
Hi, I wondered if anyone had any tips and advice for link building for mobile sites. Many thanks
Intermediate & Advanced SEO | | MarkChambers0 -
Has my site been penalized?
Our site was listed on the first page for the phrase Active SEO on Google.co.uk. We suddenly find ourselves on page 4 overnight and we're not sure what's going on. We have not undertaken an Black hat techniques however the site is fairly new. Anyone have any ideas as to what is going on?
Intermediate & Advanced SEO | | MassivePrime0 -
How Does This Site Rank So Well?!
So this website -> http://bailbondsripoffreport.com/ Ranks on the First Page for the term "Bail Bonds" It's the spammiest crappiest piece of junk website ever! lol - How does this site rank so well, it's not even a year old and it's link structure is crap. Can I like report them and have them removed lol. Any ideas would be appreciated. Thanks!
Intermediate & Advanced SEO | | utahseopros0 -
Building a mobile site.
We are building a mobile site that will be launching in another month. I’m concerned that the mobile site will start catabolizing our traditional rankings. Is there a way to keep this from happening? Should we utilize the cross domain canonical tag and point back to the traditional site URLs?
Intermediate & Advanced SEO | | SEO-Team0 -
Pages un-indexed in my site
My current website www.energyacuity.com has had most pages indexed for more than a year. However, I tried cache a few of the pages, and it looks the only one that is now indexed by Goggle is the homepage. Any thoughts on why this is happening?
Intermediate & Advanced SEO | | abernatj0