Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do you de-index and prevent indexation of a whole domain?
-
I have parts of an online portal displaying in SERPs which it definitely shouldn't be. It's due to thoughtless developers but I need to have the whole portal's domain de-indexed and prevented from future indexing.
I'm not too tech savvy but how is this achieved? No index? Robots?
thanks
-
Fastest way to do it would be through a combination of Google Webmaster Tools and Robots.txt / noindex,nofollow meta tags.
Firstly if your domain hasn't been verified within GWT, go through that process. Then proceed with the remove URL request within GWT. (Image Attached)
You will need to upload a robots.txt and/or implement the noindex meta tag within the header of the pages you no longer want indexed.
Example Robots.txt:
User-agent: *
Disallow: /offendingportal/Example Robots Meta tag:
That should be enough to get rid of the offending content from within the SERPS and prevent it from being re-indexed.
-
Hi TCE,
There are two options in my opinion for this which are the best (as their are several other options including META robots etc):
- Robots.txt: Exclude your portal via your robots.txt file.
- HTTP Auth: Let your developers add HTTP Authentication so you have to log in to see the portal. Google won't be able to see the pages itself and so will de index them.
Hope this helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Redirection of 100 domain to Main domain affects SEO?
Hi guys, An email software vendor managed by a different area of my company redirected 100 domains used for unsolicited email campaigns to my main domain. These domains are very likely to get blacklisted at some point. My SEO tool now is showing me all those domains as "linking" to my main site as do-follow links. The vendor states that this will not affect my main domain/website in any way. I'm highly concerned. I would appreciate your professional opinion about this. Thanks!!
Intermediate & Advanced SEO | | anagentile0 -
Shopify Site with Multiple Domains?
Hey there! My client has a website on Shopify. I don't even know how to open this can of worms, but let me try. The site URL is: https://mobilityequipmentforless.com/ However, there is another (older?) URL that gets updated as the main site gets updated and shows the exact same content. It's a straight duplicate, but is it's own URL and doesn't redirect to the main site. https://www.powerchairrecyclers.com/ And this isn't the SITE.Shopify back-end site name that was used for set up initially. I just have no idea what's going on here. Not sure if it's a serious error that needs to be fixed, or if it's something weird with how Shopify work. Any insight would be immensely helpful. Thanks! Mike
Intermediate & Advanced SEO | | naturalsociety0 -
One domain - Multiple servers
Can I have the root domain pointing to one server and other URLs on the domain pointing to another server without redirecting, domain masking or HTML masking? Dealing with an old site that is a mess. I want to avoid migrating the old website to the new environment. I want to work on a page by page and section by section basis, and whatever gets ready to go live I will release on the new server while keeping all other pages untouched and live on the old server. What are your recommendations?
Intermediate & Advanced SEO | | Joseph-Green-SEO0 -
Referring domain issues
Our website (blahblah).org has 32 other domains pointing to it all from the same I.P address. These domains including the one in question, were all purchased by the website owner, who has inadvertently created duplicate content and on most of these domains. Some of these referring domains have 301's, some don't - but it appears they have all been de-indexed by Google. I'm somewhat out of my depth here (most of what I've said above has come from an agency who said we should address this before being slapped by Google). However I need to explain to my line manage the actual issues in more detail and the repercussions - any anyone please offer advice please? I'm happy to use the agency, or another - but would like some second opinions if possible?
Intermediate & Advanced SEO | | LJHopkins0 -
Better to 301 or de-index 403 pages
Google WMT recently found and called out a large number of old unpublished pages as access denied errors. The pages are tagged "noindex, follow." These old pages are in Google's index. At this point, would it better to 301 all these pages or submit an index removal request or what? Thanks... Darcy
Intermediate & Advanced SEO | | 945010 -
Changing domain for a magento store
Hi all, wondering if i could gather some views on the best approach for this please... We currently have a magento site up with about 150,000 pages (although only 9k indexed in Google as product pages are set to no index by default until the default manufacturer description has been rewritten). The indexed pages are mainly category pages, filtering options and a few search results. While none of the internal pages have massive DA - seem to average about 18-24 which isn't too bad for internal pages, I guess - I would like to transfer as much of this over to the new domain. My question is, is it really feasible to have an htaccess with about 10,000 301 redirects on the current domain? The server is pretty powerful so could probably serve the file without issue but would Google be happy with that? Would it be better to use the change url option in WMT instead. Ive never used that so not sure how that would work in this cause. Would it redirect users too? As a footnote, the site is changing because of branding reasons and not because of a penalty of the site. Thanks, Carl
Intermediate & Advanced SEO | | daedriccarl0 -
Domain Forwarding - SEO Impacts?
I have a site that has been active for years - thinkbiglearnsmart.com. Awhile ago I had purchased about 50 domain names that were relevant to my company. I still have those urls and would like to use them to point to different pages on my site - just because they have good key words in the URLs. For example - one is dreamweavertrainingclassesonlinelive.com. Currently they are all redirecting to my homepage. A. is that hurting me? B. I would like to redirect to the more relevant page. ie the page dedicated to Dreamweaver training (http://thinkbiglearnsmart.com/dreamweaver-creative-cloud-training-course/ ) Will this hurt my Dreamweaver keyword for example because there is already a 301 redirect on that page from a very old Dreamweaver link which was something like thinkbiglearnsmart.com/dreamweaver C. On my hosting account where I can select where the URL forwards to - it has an option for "Location forwarding" and "Frame forwarding" - currently they are set to Frame forwarding - which one is best? Any help is much appreciated!!! Thank you!
Intermediate & Advanced SEO | | webbmason0 -
De-indexing product "quick view" pages
Hi there, The e-commerce website I am working on seems to index all of the "quick view" pages (which normally occur as iframes on the category page) as their own unique pages, creating thousands of duplicate pages / overly-dynamic URLs. Each indexed "quick view" page has the following URL structure: www.mydomain.com/catalog/includes/inc_productquickview.jsp?prodId=89514&catgId=cat140142&KeepThis=true&TB_iframe=true&height=475&width=700 where the only thing that changes is the product ID and category number. Would using "disallow" in Robots.txt be the best way to de-indexing all of these URLs? If so, could someone help me identify how to best structure this disallow statement? Would it be: Disallow: /catalog/includes/inc_productquickview.jsp?prodID=* Thanks for your help.
Intermediate & Advanced SEO | | FPD_NYC0