How is Google finding our preview subdomains?
-
I've noticed that Google is able to find, crawl and index preview subdomains we set up for new client sites (e.g. clientpreview.example.com). I know now to use "meta name="robots" and robots.txt) to block the search engines from crawling these subdomains. My question though, is how is Google finding these subdomains? We don't link to these preview domains from anywhere else, so I can't figure out how Google is even getting there.
Does anybody have any insight on this?
-
Thanks for your response Irving. We put some of our preview sites on subdomains of our main domain, but then remove them after the site goes live, so their shouldn't be any duplicate content issues. The main question is just how Google is finding these subdomains.
-
Thanks for the insight guys.
-
I don't specifically use the Google Toolbar, but others in the office may (although I don't think so). It sounds like Chrome could be a potential source as well?
-
I think that this is a good idea. But you gotta be careful.
Our competitor (who ranked #1 and we ranked at #2) had their site redesigned and the design company included the noindex on every page. They forgot to take it off when the new design went live. It took them quite a while to figure it out and we enjoyed all of their sales for about a month.
We are #1 now and they are #2. Must have been a bad design job.
-
If the subdomains are added to WMT google will know about it. if you are designing sites for clients and putting them on your site as subdomains it behooves you to make sure 100% that their dev sites are not being seen by Google. It's duplicate content and your subdomain is the original source of this content. Looks unprofessional too
a) verify any subdomain you are creating for a client in WMT
b) block it in robots.txt and noindex nofollow all pages globally
c) for the ones that are already indexed, go into google WMT and go into that subdomain account and request removal of the site in Googles index. This will remove the indexing for that subdomain only don't worry it won't remove your main site from the index.
-
I would also consider adding a noindex tag if you want the urls removed.
-
I agree with Mat. You never know, but yes Chrome could be another major source. It also depends what you set as your privacy when you setup Chrome (Send anonymous usage data to Google, Yes/No ?) and so on.
-
We usually put them behind an .htaccess login now. We've had situations where the development site have been outranking the live site. Great demo of the power of on-site optimisation, but still a bit annoying for the client.
People used to always blame google toolbar for this. Likewise using chrome could potentially add something to the "to crawl" list. I wonder what the respective privacy policies say about that. I've also seen staging sites pick up links. When an external link on the staging site has been clicked it has alerted someone else, appeared as a link back/trackback etc.
-
The discovery can be from multiple mediums. Do you or the client have Google Toolbar installed ?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What to do about this subdomain for SEO?
This is a bit of an unusual structure and I'm having difficulty explaining the question so pardon my being a 'noob', haha. The website I'm working on has some content under Forums that is hosted on another domain. The main website is https://yournorthside.org.au/ and if you select under the main Nav > Forums > Lived Experience it will take you to https://yournorthside.saneforums.org/t5/Lived-Experience-Forum/ct-p/lived-experience-forum. So it's as if it's a subdomain. (notice even the appearance of the main menu changes, weird) Apparently, saneforums.org has a requirement for that content to be on that subdomain. So therefore it's not part of my sitemap and now crawled or indexed. My question is is this structure okay? What are the implications for SEO? Should I be looking to implement some type of no follow link or something? Or is it actually beneficial in terms of all their content gives us 'link juice'? Can you link me to any resources / articles that give further insight?
Technical SEO | | kelseyc0 -
Local Google vs. default Google search
Hello Moz community, I have a question: what is the difference between a local version of Google vs. the default Google in regards to search results? I have a Mexican site that I'm trying to rank in www.google.com.mx, but my rankings are actually better if I check my keywords on www.google.com The domain is a .mx site, so wouldn't it make more sense that this page would rank higher on google.com.mx instead of the default Google site, which in theory would mean a "broader" scope? Also, what determines whether a user gets automatically directed to a local Google version vs. staying on the default one? Thanks for your valuable input!
Technical SEO | | EduardoRuiz0 -
Google Places, Plus, and National Service
I'm attempting to dive into the Google Places and Google + Local mess. First off, we are a company that services nationally. Our installer manager travels worldwide however in 15+ areas we have local sales and/or installers that service customers locally. There is no business storefront, rather they work out of their home and travel around their local area (100+ miles). Our main office is a storefront and we do service clients at our store, plus travel locally and nationally, as I mentioned. #1) Can I (should I) create separate listings for each of these installers? They are independent contractors... Use their local phone number and residential home address? Put somehow put our main 800 number as the secondary number along with our main website address? Or, should we try the bulk upload feature of 'stores' and attach them to our name and website address? #2) Our Places address is under our Marketing Director's login through Google. We have most everything else under another email address which includes our Google Plus Business page, AdWords, Merchant account, etc. Should I go back and claim the Places page through our main email account, and then merge the Places account with our Google Plus page? #3) Currently our Places page says we serve all 50 states. Should I try to keep that the same (I don't think you can select 'states' for the areas you service anymore, or can you?) or should I move to our local installers in our 15+ and growing areas? I've been reading through past Google Places discussions here and I've been trying to find out more about the bulk upload feature to see if it would work but I'm getting stumped.
Technical SEO | | WineCellarInnovations0 -
Google SERPs and NoIndex directives.
We have pages that have been added to robots.txt as url patterns in DisAllow. Also, we have the meta noindex tags on the pages themselves. But we are finding the pages in index. I don't think they are higher up in the rankings and they don't have any descriptions, any previews or any cached pages. Why does Google show these pages? Could it be due to internal or external linking?
Technical SEO | | gaganc0 -
Robots.txt for subdomain
Hi there Mozzers! I have a subdomain with duplicate content and I'd like to remove these pages from the mighty Google index. The problem is: the website is build in Drupal and this subdomain does not have it's own robots.txt. So I want to ask you how to disallow and noindex this subdomain. Is it possible to add this to the root robots.txt: User-agent: *
Technical SEO | | Partouter
Disallow: /subdomain.root.nl/ User-agent: Googlebot
Noindex: /subdomain.root.nl/ Thank you in advance! Partouter0 -
How to show ratings on Google?
One thing I have noticed recently is "review ratings" appearing in the Google search results. I have attached a screenshot which shows an example of this. I think this is a really good feature and helps make a listing stand out in the SERPs, I would certainly be more likely to click this one. My question is how do you code for it so that Google will display it? The URL of the page in question is http://www.footy-boots.com/inter-milan-away-shirt-2011-2012-9430/ 4nXyk
Technical SEO | | ukss19840 -
Blog on a subdomain vs subfolder?
Hi, Does anyone have data to show that a subfolder is better than a subdomain for a blog? From what I've read, it sounds like both are a viable option but you choose subdomain if you want to build your blog as a distinct entity. Do you get ranked more quickly with a subfolder? Do you see X% more lift? Has anyone tested or seen tests around this subject? Any input is appreciated! Thanks in advance.
Technical SEO | | sportstvjobs0 -
Google Website Optimizer
So if you are AB testing two pages: index.html and indexB.html Shouldn't I nofollow indexB.html? It has all the same content, just a different design.
Technical SEO | | tylerfraser0