Getting subdomains unindexed
-
If i turn an application off displaying a 503 error will that get my site unindexed from search engines?
-
Subdomains can be verified as their own site in GWT. Verify the subdomain in GWT, then put a robots.txt on that subdomain excluding the entire subdomain, then request removal in GWT of that entire subdomain. I've had to remove staging and dev sites a couple of times myself.
A couple of things I've found useful in this situation is to make the robots.txt files for both the dev and live sites read only, so you don't accidentally overwrite one with the other when pushing a site live. You can also sign up for a free tool like Pole Position's Code Monitor that will look at the code of a page (including your robots.txt url) once a day and email you if there are any changes so you can fix the file then go hunt down whoever changed the file.
-
GWT was the first placed i checked unfortunately you can only remove directories or pages. I need entire subdomained sites to be removed (in fact they shouldn't of been indexed in the first place).
We use subdomains for our development testing environment when creating client sites and once the site is approved we push it live replacing the old site. Somehow these testing sites are getting indexed and it may pose a threat to duplicate content on different domains. So i am trying to find a solution to get the subdomains (100's of them) unindexed.
I understand a 301 redirect is best but that isn't really applicable since these test sites still need to be reached by clients.
-
With a robots.txt blocking it, you can then go into Google Webmaster Tools and request removal of that particular page or folder from Google's index.
-
No index tag on it works, and putting up a robots.txt that disallows everyone should work as well.
-
Thanks for the quick reply, i will have to try that. Essentially i am trying to get the site un-indexed but i wasn't sure if a 503 would do the trick.
-
Eventually, but that's the code Google recommends to return when your site is having downtime, so I would expect them to be more lenient towards not removing things right away. I wouldn't expect it to be as efficient as returning a 404 or a 410.
The best way to get content de-indexed is to return a page with a meta noindex tag on it, if you're really keen on getting it removed immediately.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Getting 'Indexed, not submitted in sitemap' for around a third of my site. But these pages ARE in the sitemap we submitted.
As in the title, we have a site with around 40k pages, but around a third of them are showing as "Indexed, not submitted in sitemap" in Google Search Console. We've double-checked the sitemaps we have submitted and the URLs are definitely in the sitemap. Any idea why this might be happening? Example URL with the error: https://www.teacherstoyourhome.co.uk/german-tutor/Egham Sitemap it is located on: https://www.teacherstoyourhome.co.uk/sitemap-subject-locations-surrey.xml
Technical SEO | | TTYH0 -
301 Redirects from Subdomain to Location Pages
I have a client site that is getting redesigned. Its a multi location service provider. Currently (for whatever reason) the location pages are sub domains. https://<location-name>.site.com/ In the new design the locations will be on the main domain. https://site.com/locations/<location-name> We are considering using 301 redirects from the current sub domains to the new location pages on the main domain. The current sub domains are setup on a multi-site with A records for each one in our GoDaddy account. Would like to get feedback on any unforeseen SEO issues that anyone might have input on.
Technical SEO | | ColeBField12210 -
Impact of Medium blog hosted on my subdomain
I am using the Medium blogging platform to blog, but it is pointed to my site and appears at blog.mysite.com. Since the content is hosted on Medium and pointed to my subdomain via an A Record / CNAME / etc... 1. Will my domain get credit for backlinks to the blog content? 2. If Medium changes in the future and no longer points to my subdomain, will I lose all of the backlinks I've built up?
Technical SEO | | davidevans_seo0 -
How can I get Google to forget an https version of one page on my site?
Google mysteriously decided to index the broken, https version of one page on my company's site (we have a cert for the site, but this page is not designed to be served over https and the CSS doesn't load). The page already has many incoming links to the http version, and it has a canonical URL with http. I resubmitted it on http with webmaster tools. Is there anything else I could do?
Technical SEO | | BostonWright0 -
Sill cant seem to get images indexed by Goolge
Hi I've been trying for months now to get our images indexed by Google Would love to know if anyone can see a reason why they haven't here is the sitemap http://www.tidy-books.com/sitemapseosuite/US/sitemapUS2.xml Also 54 pages were submitted but only 9 have been indexed? Any help would be great Thanks
Technical SEO | | tidybooks0 -
Why do I get duplicate content errors just for tags I place on blog entries?
I the SEO MOZ crawl diagnostics for my site, www.heartspm.com, I am getting over 100 duplicate content errors on links built from tags on blog entries. I do have the original base blog entry in my site map not referencing the tags. Similarly, I am getting almost 200 duplicate meta description errors in Google Webmaster Tools associated with links automatically generated from tags on my blog. I have more understanding that I could get these errors from my forum, since the forum entries are not in the sitemap, but the blog entries are there in the site map. I thought the tags were only there to help people search by category. I don't understand why every tag becomes its' own link. I can see how this falsely creates the impression of a lot of duplicate data. As seen in GWT: Pages with duplicate meta descriptions Pages [
Technical SEO | | GerryWeitzCustomer concerns about the use of home water by pest control companies.](javascript:dropInfo('zip_0div', 'none', document.getElementById('zip_0zipimg'), 'none', null);)
/category/job-site-requirements
/tag/cost-of-water
/tag/irrigation-usage
/tag/save-water
/tag/standard-industry-practice
/tag/water-use 6 [
Pest control operator draws analogy between Children's Day and the state of the pest control industr](javascript:dropInfo('zip_1div', 'none', document.getElementById('zip_1zipimg'), 'none', null);)
/tag/children-in-modern-world
/tag/children
/tag/childrens-day
/tag/conservation-medicine
/tag/ecowise-certified
/tag/estonia
/tag/extermination-service
/tag/exterminator
/tag/green-thumb
/tag/hearts-pest-management
/tag/higher-certification
/tag/higher-education
/tag/tartu
/tag/united-states
0 -
How do I get rid of irrelevant back links pointing to missing pages on my site
Hi all, My site was hacked about a year ago and as a result I now have a ton of back links from irrelevant sites pointing to pages on my site that no longer exist. The followed back links section on the Competitive domain analysis tool shows about 3 pages worth of these horrible links. I have 2 questions: how bad is this for my site's SEO (which isn't good anyway, Page Rank 0) and how do I get rid of them? Any help would be much appreciated. Thanks, Andy WkXz0
Technical SEO | | getzen560 -
Snagged an Expired Domain Best way to get all the Link Juce?
Found a PR4 domain that had expired and not been renewed. Best way to get all the link juice from it? just 301 the whole thing to MY main domain?
Technical SEO | | bozzie3110