Bloking pages in roborts.txt that are under a redirected subdomain

[RaquelSaiz]

Hi Everyone,

I have a lot of Marketo landing pages that I don't want to show in SERP. Adding the noindex meta tag for each page will be too much, I have thousands of pages.

Blocking it in roborts.txt could have been an option, BUT, the subdomain homepage is redirected to my main domain (with a 302) so I may confuse search engines ( should they follow the redirect or should they block)

marketo.mydomain.com is redirected to www.mydomain.com

disallow: / (I think this will be confusing with the redirect)

I don't have folders, all pages are under the subdomain, so I can't block folders in Robots.txt also

Would anyone had this scenario or any suggestions?

I appreciate your thoughts here.

Thank you

Rachel

[RaquelSaiz]

Thank you so much for you answer!

the home page in the subdomain is redirected but none of the actual pages in the subdomain are, and because there are so many of them, it would be easier to block them in robots.txt, even if there is small change that Google will still index them. But because the home page is redirected, I don't want to confuse Google with a Disallow: /

Could I do Disallow: / and then Allow: /homepage.html

[effectdigital]

Under usual circumstances, Google won't index redirecting addresses which it considers to be a shallow form of 'doorway' page (Google doesn't like to rank those). If I am reading your post right, no pages on the redirected sub-domain can be visited as they all now redirect. Google should start dropping those old URLs from its index automatically. It's important to note that you should be using 301s to eventually de-index the old URLs. If you're using 302s then you're telling Google that the old pages are only being redirected temporarily and they will return (which could mess with indexation)

If I am reading your post wrong and some pages are still live on the old subdomain and are not redirecting, and you want Google to redirect most addresses but some are staying (for whatever reason) and you still want to de-index those specific ones, Meta no-index (sorry) really is your best bet! Robots.txt tells Google not to crawl a page but it doesn't tell Google not to index a URL if external metrics (inbound links) are strong enough. Both are 'directives' and Google is forced to obey neither

I think from what you are saying, I'd just leave the redirects and let Google do its work. Make sure they're 301s, though. You're right that Robots.txt might end up confusing things. Robots.txt can also sometimes 'kill' the SEO authority of a page. If you did that for all the redirecting pages, no equity would flow through your 301s (hazardous)