How do you de-index and prevent indexation of a whole domain?
-
I have parts of an online portal displaying in SERPs which it definitely shouldn't be. It's due to thoughtless developers but I need to have the whole portal's domain de-indexed and prevented from future indexing.
I'm not too tech savvy but how is this achieved? No index? Robots?
thanks
-
Fastest way to do it would be through a combination of Google Webmaster Tools and Robots.txt / noindex,nofollow meta tags.
Firstly if your domain hasn't been verified within GWT, go through that process. Then proceed with the remove URL request within GWT. (Image Attached)
You will need to upload a robots.txt and/or implement the noindex meta tag within the header of the pages you no longer want indexed.
Example Robots.txt:
User-agent: *
Disallow: /offendingportal/Example Robots Meta tag:
That should be enough to get rid of the offending content from within the SERPS and prevent it from being re-indexed.
-
Hi TCE,
There are two options in my opinion for this which are the best (as their are several other options including META robots etc):
- Robots.txt: Exclude your portal via your robots.txt file.
- HTTP Auth: Let your developers add HTTP Authentication so you have to log in to see the portal. Google won't be able to see the pages itself and so will de index them.
Hope this helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it worth keeping a decades-old domain that's merely 301 redirecting to the main domain?
Hi fellow Moz SEOs, We have a bigger client who we just did an SEO Site Audit for, and it was discovered that they have several domain names that are simply 301 redirecting to their main domain name. One of their domains in particular is decades old, and the client is asking if there is any value in keeping it (and the others), or simply leaving them as-is. Considering the domain age, does anyone have any recommendations? Much appreciated, Zack Barton
Intermediate & Advanced SEO | | Zack
Barton Interactive
(833) 442.6853 // office
(408) 910.7750 // mobile
https://bartoninteractive.com0 -
New domain or subdirectory?
I noticed my domain authority has dropped slightly in the recent update, and it has me re-thinking a strategy for a website I just recently launched. I purchased the domain name kansasisbeautiful.com about a year ago and have been working on building it for most of that time. Earlier in August, I went ahead and launched it. However, towards the end of the development of the website, I decided to just put it in a subdirectory of my parent company (my photography business) at mickeyshannon.com/kansas and redirected the kansasisbeautiful.com domain to the subdirectory. mickeyshannon.com is my photography business website. The Kansas website has it's own distinct design, but is powered completely by my photography. I created it for a few purposes, including promoting tourism to the state of Kansas and to publish a book on Kansas travel next year, but one of it's main goals is also to help sell my photography prints. I decided to put it in a subdirectory (mickeyshannon.com/kansas) as I had hoped it might drive more traffic into buying photo prints if it lived on my main website. However, I've been re-thinking my strategy and have been wondering if I'm competing against myself too much. Many of my photography prints have the name of a location in them and have their own URL per photo (for example: "Flint Hills Spring Sunrise" is at http://www.mickeyshannon.com/photo/flint-hills-spring-sunset/). It makes me wonder if the new Kansas travel website page for the Flint Hills (http://www.mickeyshannon.com/kansas/flint-hills/) is competing for that keyword. Would I be better moving mickeyshannon.com/kansas to kansasisbeautiful.com? I was worried having so many backlinks back to my photography site would send up red flags with Google as if the kansasisbeautiful.com website was just a spammy website created to push traffic to mickeyshannon.com when it really has it's own purpose. Any thoughts on whether using the domain name or keeping it at the subdomain level is better? Hopefully that made sense. Thanks, Mickey
Intermediate & Advanced SEO | | VSphoto0 -
Google not Indexing images on CDN.
My URL is: http://bit.ly/1H2TArH We have set up a CDN on our own domain: http://bit.ly/292GkZC We have an image sitemap: http://bit.ly/29ca5s3 The image sitemap uses the CDN URLs. We verified the CDN subdomain in GWT. The robots.txt does not restrict any of the photos: http://bit.ly/29eNSXv. We used to have a disallow to /thumb/ which had a 301 redirect to our CDN but we removed both the disallow in the robots.txt as well as the 301. Yet, GWT still reports none of our images on the CDN are indexed. The above screenshot is from the GWT of our main domain.The GWT from the CDN subdomain just shows 0. We did not submit a sitemap to the verified subdomain property because we already have a sitemap submitted to the property on the main domain name. While making a search of images indexed from our CDN, nothing comes up: http://bit.ly/293ZbC1While checking the GWT of the CDN subdomain, I have been getting crawling errors, mainly 500 level errors. Not that many in comparison to the number of images and traffic that we get on our website. Google is crawling, but it seems like it just doesn't index the pictures!? Can anyone help? I have followed all the information that I was able to find on the web but yet, our images on the CDN still can't seem to get indexed.
Intermediate & Advanced SEO | | alphonseha0 -
How to switch brand domain and address previous use of domain
We recently acquired a new domain to replace existing as it better fits our brand. We have little/no organic value on existing domain so switching is not an issue. However the newly acquired domain was previously used in a different industry and has inbound links with significant spam scores. How can we let Google know that these links are not valid for our business and start rebuilding reputation of the domain? Disavow tool?
Intermediate & Advanced SEO | | Marlette0 -
Domain Authority
Hi Our website Domain Authority isn't as high and was wondering why it's not increasing. Compared to 1 or 2 competitors we're not scoring as high as them. Are rankings are good for all chosen keywords. Just trying to get a better handle where our site is falling short on.
Intermediate & Advanced SEO | | Cocoonfxmedia1 -
Google Indexed my Site then De-indexed a Week After
Hi there, I'm working on getting a large e-commerce website indexed and I am having a lot of trouble.
Intermediate & Advanced SEO | | Travis-W
The site is www.consumerbase.com. We have about 130,000 pages and only 25,000 are getting indexed. I use multiple sitemaps so I can tell which product pages are indexed, and we need our "Mailing List" pages the most - http://www.consumerbase.com/mailing-lists/cigar-smoking-enthusiasts-mailing-list.html I submitted a sitemap a few weeks ago of a particular type of product page and about 40k/43k of the pages were indexed - GREAT! A week ago Google de-indexed almost all of those new pages. Check out this image, it kind of boggles my mind and makes me sad. http://screencast.com/t/GivYGYRrOV While these pages were indexed, we immediately received a ton of traffic to them - making me think Google liked them. I think our breadcrumbs, site structure, and "customers who viewed this product also viewed" links would make the site extremely crawl-able. What gives?
Does it come down to our site not having enough Domain Authority?
My client really needs an answer about how we are going to get these pages indexed.0 -
Using a .de domain for non-german website. Sane?
Hi there guys(and girls), I bought a beautiful and clever .de domain, however I plan to use it for a brazillian website. Since GWT does not allow me to change the target location, and I am stuck with Germany on Google's eyes, I would like to know how bad would it be to stick with the .de domain? My content won't be international. It will focus only in portuguese(Brazil) content. Is there a way to make it work? I would really like to stick with this domain, but Google's organic traffic cannot be overlooked... Thanks in advance for all input I can get. Cheers
Intermediate & Advanced SEO | | adepalma0 -
Which domain should I use?
I own a couple domains that are specific to a product and would like to know which one folks on here recommend. Primary Google Search Term Example: "Tax Bond" Example Domain 1: www.taxbonds.net Example Domain 2: www.tax-bond.net I've done research on here before and have come to the conclusion that hyphenated domains aren't bad (no more than 2 hyphens though). So, do I go for the EXACT search term with the hyphen or do I go for the pluralized search term without the hyphen, even though most people will not add the "s" in the Google search? Thanks, Alex
Intermediate & Advanced SEO | | dbuckles0