Getting subdomains unindexed
-
If i turn an application off displaying a 503 error will that get my site unindexed from search engines?
-
Subdomains can be verified as their own site in GWT. Verify the subdomain in GWT, then put a robots.txt on that subdomain excluding the entire subdomain, then request removal in GWT of that entire subdomain. I've had to remove staging and dev sites a couple of times myself.
A couple of things I've found useful in this situation is to make the robots.txt files for both the dev and live sites read only, so you don't accidentally overwrite one with the other when pushing a site live. You can also sign up for a free tool like Pole Position's Code Monitor that will look at the code of a page (including your robots.txt url) once a day and email you if there are any changes so you can fix the file then go hunt down whoever changed the file.
-
GWT was the first placed i checked unfortunately you can only remove directories or pages. I need entire subdomained sites to be removed (in fact they shouldn't of been indexed in the first place).
We use subdomains for our development testing environment when creating client sites and once the site is approved we push it live replacing the old site. Somehow these testing sites are getting indexed and it may pose a threat to duplicate content on different domains. So i am trying to find a solution to get the subdomains (100's of them) unindexed.
I understand a 301 redirect is best but that isn't really applicable since these test sites still need to be reached by clients.
-
With a robots.txt blocking it, you can then go into Google Webmaster Tools and request removal of that particular page or folder from Google's index.
-
No index tag on it works, and putting up a robots.txt that disallows everyone should work as well.
-
Thanks for the quick reply, i will have to try that. Essentially i am trying to get the site un-indexed but i wasn't sure if a 503 would do the trick.
-
Eventually, but that's the code Google recommends to return when your site is having downtime, so I would expect them to be more lenient towards not removing things right away. I wouldn't expect it to be as efficient as returning a 404 or a 410.
The best way to get content de-indexed is to return a page with a meta noindex tag on it, if you're really keen on getting it removed immediately.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site indexed by Google, but (almost) never gets impressions
Hi there, I have a question that I wasn't able to give it a reasonable answer yet, so I'm going to trust on all of you. Basically a site has all its pages indexed by Google (I verified with site:sitename.com) and it also has great and unique content. All on-page grades are A with absolutely no negative factors at all. However its pages do not get impressions almost at all. Of course I didn't expect it to be on page 1 since it has been launched on Dec, 1st, but it looks like Google is ignoring (or giving it bad scores) for some reason. Only things that can contribute to that could be: domain privacy on the domain, redirect from the www to the subdomain we use (we did this because it will be a multi-language site, so we'll assign to each country a subdomain), recency (it has been put online on Dec 1st and the domain is just a couple of months old). Or maybe because we blocked crawlers for a few days before the launch? Exactly a few days before Dec 1st. What do you think? What could be the reason for that? Thanks guys!
Technical SEO | | ruggero0 -
Struggling to get indexed and ranked
I am working on a brand new website and really struggling to get the site indexed and listed for it's business name! I don't normally struggle and have got clients ranked for much more challenging keywords so I'm a bit stuck! The site is a new domain and has been live for about two months. The business previously used an old domain and this has been correctly 301'd to the new domain. There is no duplicate content with any other website when checked through Copyscape.com Webmaster Tools has been set-up and verified and this shows the site is being crawled but in Google site:www.website.com.au shows no pages as being indexed. Google Places has been set-up and verified, the site has also been added to local citation sites. There are also a few incoming links from other sources. Robots.txt is fine and has been checked. Business name mentioned in the title tag, footer and throughout the site. Does anyone have any ideas how I might be able to get ranked or is it just a waiting game? Or have I missed out something really obvious?? My last step is doing a crawl test to see if this brings up anything I have missed. Thanks Karen
Technical SEO | | Karen_Dauncey0 -
URLs in Greek, Greeklish or English? What is the best way to get great ranking?
Hello all, I am Greek and I have a quite strange question for you. Greek characters are generally recognized as special characters and need to have UTF-8 encoding. The question is about the URLs of Greek websites. According the advice of Google webmasters blog we should never put the raw greek characters into the URL of a link. We always should use the encoded version if we decide to have Greek characters and encode them or just use latin characters in the URL. Having Greek characters un-encoded could likely cause technical difficulties with some services, e.g. search engines or other url-processing web pages. To give you an example let's look at A) http://el.wikipedia.org/wiki/%CE%95%CE%BB%CE%B2%CE%B5%CF%84%CE%AF%CE%B1which is the URL with the encoded Greek characters and it shows up in the browser asB) http://el.wikipedia.org/wiki/Ελβετία The problem with A is that everytime we need to copy the URL and paste it somewhere (in an email, in a social bookmark site, social media site etc) the URL appears like the A, plenty of strange characters and %. This link sometimes may cause broken link issues especially when we try to submit it in social networks and social bookmarks. On the other hand, googlebot reads that url but I am wondering if there is an advantage for the websites who keep the encoded URLs or not (in compairison to the sites who use Greeklish in the URLs)! So the question is: For the SEO issues, is it better to use Greek characters (encoded like this one http://el.wikipedia.org/wiki/%CE%95%CE%BB%CE%B2%CE%B5%CF%84%CE%AF%CE%B1) in the URLs or would it be better to use just Greeklish (for example http://el.wikipedia.org/wiki/Elvetia ? Thank you very much for your help! Regards, Lenia
Technical SEO | | tevag0 -
Getting a complete list of all incoming links
I'm trying to clean up my spam incoming links and can't find where to download a complete list. Google Webmaster Tools has a button to download "Sample Links" or "Latest Links" but they don't show them all. Looking for wisdom from all the gurus out there. Thanks, Jean
Technical SEO | | JeanYates0 -
Trying to get on Google page one for keyword "criminal defense attorney san diego". What can I do?
I'm trying to help a friend who is an attorney get on page one for the keyword "criminal defense attorney san diego." So far I've changed his title and description tags since they weren't optimized before. (SERP shows old title tag, however I submitted a XML sitemap through Webmaster tools to get the new title tags updated.) He also had a few duplicate pages, but I took care of that with some 301 redirects. I also added a h1 tag, alt image tag, and more content. I also spent a few hours building links for him. He currently has a page authority of 52 and domain authority of 44 with a decent amount of links pointing to his site. I'm wondering why he's stuck on page 4, when his competitors that have less impressive numbers seem to show up on page 1. I did look at his link profile using OSE and I'm worried that his old SEO guy got him spam links. His website is www.nasserilegal.com, however the page I was focusing on was www.nasserilegal.com/criminal.html Any advice would be great.
Technical SEO | | micasalucasa0 -
How do you get a Google+ pic in your SERP snippet
Hi from from 20 degrees C 83% humidity wetherby UK 🙂 A few weeks back i decided i needed to get my pretty face appearing in my serps for www.davidclick.com But after having set up a Gppgle+ account and linking my site to the Google+ account i think I may have done something wrong 😞 I linked to the Google+ page via a footer link in www.davidclick.com but alas I'm not able to get my face in my SERP which this website has: http://i216.photobucket.com/albums/cc53/zymurgy_bucket/google-plus-picJPGcopy.jpg So my question is please - "How do you get your Google+ account image to appear in the SERPS. Ta muchly,
Technical SEO | | Nightwing
David0 -
How to block google robots from a subdomain
I have a subdomain that lets me preview the changes I put on my site. The live site URL is www.site.com, working preview version is www.site.edit.com The contents on both are almost identical I want to block the preview version (www.site.edit.com) from Google Robots, so that they don't penalize me for duplicated content. Is it the right way to do it: User-Agent: * Disallow: .edit.com/*
Technical SEO | | Alexey_mindvalley0 -
SEO-MOZ bar question on root vs subdomain / canonicalization issues
When I look at the SEO-MOZ bar for our site and click next to subdomain (# links from #domains) it shows my main incoming links etc. but when I click on root domain ity shows mydomain/default.asp and 4 incoming links as well as a message that says this url redirects to another url. Does this imply canonicalization issues or is there a 301 redirect to my non /default.asp correcting this issue. Thanks kindly, Howard
Technical SEO | | mrkingsley0