Bloking pages in roborts.txt that are under a redirected subdomain
-
Hi Everyone,
I have a lot of Marketo landing pages that I don't want to show in SERP. Adding the noindex meta tag for each page will be too much, I have thousands of pages.
Blocking it in roborts.txt could have been an option, BUT, the subdomain homepage is redirected to my main domain (with a 302) so I may confuse search engines ( should they follow the redirect or should they block)
marketo.mydomain.com is redirected to www.mydomain.com
disallow: / (I think this will be confusing with the redirect)
I don't have folders, all pages are under the subdomain, so I can't block folders in Robots.txt also
Would anyone had this scenario or any suggestions?
I appreciate your thoughts here.
Thank you
Rachel
-
Thank you so much for you answer!
the home page in the subdomain is redirected but none of the actual pages in the subdomain are, and because there are so many of them, it would be easier to block them in robots.txt, even if there is small change that Google will still index them. But because the home page is redirected, I don't want to confuse Google with a Disallow: /
Could I do Disallow: / and then Allow: /homepage.html
-
Under usual circumstances, Google won't index redirecting addresses which it considers to be a shallow form of 'doorway' page (Google doesn't like to rank those). If I am reading your post right, no pages on the redirected sub-domain can be visited as they all now redirect. Google should start dropping those old URLs from its index automatically. It's important to note that you should be using 301s to eventually de-index the old URLs. If you're using 302s then you're telling Google that the old pages are only being redirected temporarily and they will return (which could mess with indexation)
If I am reading your post wrong and some pages are still live on the old subdomain and are not redirecting, and you want Google to redirect most addresses but some are staying (for whatever reason) and you still want to de-index those specific ones, Meta no-index (sorry) really is your best bet! Robots.txt tells Google not to crawl a page but it doesn't tell Google not to index a URL if external metrics (inbound links) are strong enough. Both are 'directives' and Google is forced to obey neither
I think from what you are saying, I'd just leave the redirects and let Google do its work. Make sure they're 301s, though. You're right that Robots.txt might end up confusing things. Robots.txt can also sometimes 'kill' the SEO authority of a page. If you did that for all the redirecting pages, no equity would flow through your 301s (hazardous)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Redirecting Homepage to Subdomain Bad or Good Idea??
I have a very old forum that still gets a lot of traffic, but when migrating over to another software that is cloud based we cannot redirect using same domain, SO the only option would to be to change the cname on a subdomain and then REDIRECT all the traffic from the ROOT domain permanently - would this be a bad move as the root domain wouldnt be used anymore as now its just setup to be redirected in order to use the software we need to use? Domain is 17 years old.
Technical SEO | | vbsk0 -
Help Center/Knowledgebase effects on SEO: Is it worth my time fixing technical issues on no-indexed subdomain pages?
We're a SaaS company and have a pretty extensive help center resource on a subdomain (help.domain.com). This has been set up and managed over a few years by someone with no knowledge of SEO, meaning technical things like 404 links, bad redirects and http/https mixes have not been paid attention to. Every page on this subdomain is set to NOT be indexed in search engines, but we do sometimes link to help pages from indexable posts on the main domain. After spending time fixing problems on our main website, our site audits now flag almost solely errors and issues on these non-indexable help center pages every week. So my question is: is it worth my time fixing technical issues on a help center subdomain that has all its pages non-indexable in search engines? I don't manage this section of the site, and so getting fixes done is a laborious process that requires going through someone else - something I'd rather only do if necessary.
Technical SEO | | mglover19880 -
My homepage redirects to itself?
Hi there - I'm not a SEO so help would be appreciated! Moz is telling me we have a redirect loop but the URLs are the same. https://www.example.com/ to https://www.example.com/ Why is my homepage creating a redirect loop to itself? We use Wordpress and I do not have any redirects listed for our homepage. Could this have something to do with switching to https in April? Thanks, Katherine
Technical SEO | | kmmartin0 -
I need help with redirecting chain to another page and 301, I don't understand on how to fix
Redirect Chain <label>What it is:</label> Your page is redirecting to a page that is redirecting to a page that is redirecting to a page... and so on. Learn more about redirection best practices. <label>Why it's an issue:</label> Every redirect hop loses link equity and offers a poor user experience, which will negatively impact your rankings. <label>How to fix it:</label> Chiaryn says: “Redirect chains are often caused when multiple redirect rules pile up, such as redirecting a 'www' to non-www URL or a non-secure page to a secure/https: page. Look for any recurring chains that could be rewritten as a single rule. Be particularly careful with 301/302 chains in any combination, as the 302 in the mix could disrupt the ability of the 301 to pass link equity.” This is not helping me I don't understand about the 301 do I use the www.jasperartisanjewelry.com or the /jasperartisanjewelry.com I'm confused
Technical SEO | | geanmitch0 -
Redirecting root domain to a page based on user login
We have our main URL redirecting non-logged in users to a specific page and logged in users are directed to their dashboard when going to the main URL. We find this to be the most user-friendly, however, this is all being picked up as a 302 redirect. I am trying to advise on the ideal way to accomplish this, but I am not having much luck in my search for information. I believe we are going to put a true homepage at the root domain and simply redirect logged in users as usual when they hit the URL, but I'm still concerned this will cause issues with Google and other search engines. Anyone have experience with domains that need to work in this manner? Thank you! Anna
Technical SEO | | annalytical0 -
Duplicate pages
Hi Can anyone tell me why SEO MOZ thinks these paes are duplicates when they're clearly not? Thanks very much Kate http://www.katetooncopywriter.com.au/how-to-be-a-freelance-copywriter/picture-1-58/ http://www.katetooncopywriter.com.au/portfolio/clients/other/ http://www.katetooncopywriter.com.au/portfolio/clients/travel/ http://www.katetooncopywriter.com.au/webservices/what-i-do/blog-copywriter/
Technical SEO | | ToonyWoony0 -
Can I do a redirect to a new domain name only a couple of weeks after having redirected to another domain?
I have a client with two website with very similar content. Both had a lot of inbound links and performed fairly well in SERPS. We recently combined both sites and have redirected one of the domains to the other. The traffic dipped slightly initially, but is recovering nicely. Now the client registered a new domain name he would like to use for the site. Should I wait a few weeks for everything to settle down after the first redirect/consolidation of sites before doing a new redirect to a new domain name, or should I not worry about having any issues with doing it right away?
Technical SEO | | Drewco0 -
Renaming of pages
About 2 months ago one of our clients renamed a section of his website. The worst part is that the URLs of the page also changed. New page: http://www.meresverige.dk/rejser/malmo Old page: http://www.meresverige.dk/rejser/malmoe The problem now is that the new page get absolutely no page-rank transfered from the old page. It also get no mozrank at all. Also if I try to find it in the Open Site Explorer it can not be found.The old page can, but not the new one. We have updated the sitemap.xml and also done proper 301 redirect for the pages since about 2 months. Any ideas here? This page was a very important page in terms of traffic so very much thankful for any input. Have a great day Fredrik
Technical SEO | | Resultify0