I am looking for best way to block a domain from getting indexed ?
-
We have a website http://www.example.co.uk/ which leads to another domain (https://online.example.co.uk/) when a user clicks,in this case let us assume it to be Apply now button on my website page. We are getting meta data issues in crawler errors from this (https://online.example.co.uk/) domain as we are not targeting any meta content on this particular domain. So we are looking to block this domain from getting indexed to clear this errors & does this effect SERP's of this domain (**https://online.example.co.uk/) **if we use no index tag on this domain.
-
Hi the real challenge here is we are not using any Google entities like webmaster tools etc on https://online.example.co.uk so to my knowledge robots.txt wont work, will be waiting to hear from you if we have any other options here.
Thanks
P
-
I'd recommend putting a robots.txt file on the https://online.example.co.uk site you don't want indexed.
Just save the following as robots.txt:
User-agent: *
Disallow: /then add it to the the root folder of the site like https://online.example.co.uk/robots.txt
This tells all the search spiders not to index or crawl any pages on the entire site.
-
Hi thanks for letting me know this , it will be great if you have any wildcard to solve this
-
If you use the meta noindex on page then it will be blocked from the index. So, yes, it will effect the SERPs of that domain by removing any results from the SERPs.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a way to get a list of all pages of your website that are indexed in Google?
I am trying to put together a comprehensive list of all pages that are indexed in Google and have differing opinions on how to do this.
Technical SEO | | SpodekandCo0 -
Changed domains, saw significant drop in domain authority
Hello, A little over a month ago we switched our domain from A to B. The original was .com and the new one is at a .tech for good measure. We have meticulously done 301 redirects page by page but there has been a massive drop in traffic and I checked here to see that we've gone from a 30 domain authority to a 9. The company is a few months over 1 year old, but we're really looking at traffic accumulated in just about 8 months for the old domain. Is there any way to recover some of the old juice in this one other than the re-directs? Our number of backlinks have also severely dropped despite re-directs.
Technical SEO | | SteveSaf0 -
Why are my webpages not getting indexed?
I want to figure out why a lot of my pages for my website are not getting indexed by google. I have installed the SEO plugin by Yoast to my wordpress website. Under the titles and meta section of the plugin options I have set categories and tags to noindex. In WMT, google is saying that all my category pages and most of my tag pages are not being indexed. I want to make sure that the reason these pages are not being indexed are because of the SEO plugin. I want to prevent duplicate content so that is the reason I have set my categories and tags to noindex. Please respond if you know the absolute answer, its very important that I have my website indexed the proper way I want it to.
Technical SEO | | Dino640 -
What is the best way to refresh a webpage of a news site, SEO wise?
Hello all, we have a client which is a sports website. In fact it is a veyr big website and has a huge number of news per day. This is mostly the reason why it refreshes some of its pages with news list every 420 seconds. We currently use meta refresh. I have read here and elsewhere that meta refreshes should be avoided. But we don't do it to send to another page and pass any kind of page authority / juice. Is in this case javascript refresh better? Is there any other better way. What do you think & suggest? Thank you!
Technical SEO | | pkontopoulos0 -
What's the best way to handle Overly Dynamic Url's?
So my question is What the best way to handle Overly Dynamic Url's. I am working on a real estate agency website. They are selling/buying properties and the url is as followed. ttp://www.------.com/index.php?action=calculator&popup=yes&price=195000
Technical SEO | | Angelos_Savvaidis0 -
What is the best way to fix legacy overly-nested URLs?
Hi everyone, Due to some really poor decisions I made back when I started my site several years ago, I'm lumbered with several hundred pages that have overly-nested URLs. For example: /theme-parks/uk-theme-parks/alton-towers/attractions/enterprise I'd prefer these to feature at most three layers of nesting, for example: /reviews/alton-towers/enterprise Is there a good approach for achieving this, or is it best just to accept the legacy URLs as an unfixable problem, and make sure that future content follows the new structure? I can easily knock together a script to update the aliases for the existing content, but I'm concerned about having hundreds of 301 redirects (could this be achieved with a single regular express in .htaccess, for example?). Any guidance appreciated. Thanks, Nick
Technical SEO | | ThemeParkTourist0 -
Sub Domains
Hi,,, Okay we have 1 main site , a few years back we went down the road of sub domains and generated about 10. They have page rank and age but we wish to move them back to the main web site. What is the correct or best way to achieve this. 1 copy all content to the main web site creating dup pages and then use a redirects from the sub pages to the new dup pages on the main domain... or 2 write new content on the main domain for the subdomain pages and redirect to the new content. Problem with 2 is the amount of work involved...
Technical SEO | | NotThatFast0 -
What is best practice for redirecting "secondary" domain names?
For sites with multiple top-level domains that have been secured for a business or organization, I'm curious as to what is considered best practice for setting up 301 redirects for secondary domains. Is it best to do the 301 redirects at the registrar level, or the hosting level? So that .net, .biz, or other secondary domains funnel visitors to the correct primary/main domain name. I'm looking for the "best practice" answer and want to avoid duplicate content problems, or penalties from the search engines. I'm not trying to game the system with dozens of domain names, simply the handful of domains that are important to the client. I've seen some registrars recommend hosting secondary domains, and doing redirects from the hosting level (and they use meta refresh for "domain forwarding," which I want to avoid). It seems rather wasteful to set up hosting for a secondary domain and then 301 each URL.
Technical SEO | | Scott-Thomas0