Bloking pages in roborts.txt that are under a redirected subdomain
-
Hi Everyone,
I have a lot of Marketo landing pages that I don't want to show in SERP. Adding the noindex meta tag for each page will be too much, I have thousands of pages.
Blocking it in roborts.txt could have been an option, BUT, the subdomain homepage is redirected to my main domain (with a 302) so I may confuse search engines ( should they follow the redirect or should they block)
marketo.mydomain.com is redirected to www.mydomain.com
disallow: / (I think this will be confusing with the redirect)
I don't have folders, all pages are under the subdomain, so I can't block folders in Robots.txt also
Would anyone had this scenario or any suggestions?
I appreciate your thoughts here.
Thank you
Rachel
-
Thank you so much for you answer!
the home page in the subdomain is redirected but none of the actual pages in the subdomain are, and because there are so many of them, it would be easier to block them in robots.txt, even if there is small change that Google will still index them. But because the home page is redirected, I don't want to confuse Google with a Disallow: /
Could I do Disallow: / and then Allow: /homepage.html
-
Under usual circumstances, Google won't index redirecting addresses which it considers to be a shallow form of 'doorway' page (Google doesn't like to rank those). If I am reading your post right, no pages on the redirected sub-domain can be visited as they all now redirect. Google should start dropping those old URLs from its index automatically. It's important to note that you should be using 301s to eventually de-index the old URLs. If you're using 302s then you're telling Google that the old pages are only being redirected temporarily and they will return (which could mess with indexation)
If I am reading your post wrong and some pages are still live on the old subdomain and are not redirecting, and you want Google to redirect most addresses but some are staying (for whatever reason) and you still want to de-index those specific ones, Meta no-index (sorry) really is your best bet! Robots.txt tells Google not to crawl a page but it doesn't tell Google not to index a URL if external metrics (inbound links) are strong enough. Both are 'directives' and Google is forced to obey neither
I think from what you are saying, I'd just leave the redirects and let Google do its work. Make sure they're 301s, though. You're right that Robots.txt might end up confusing things. Robots.txt can also sometimes 'kill' the SEO authority of a page. If you did that for all the redirecting pages, no equity would flow through your 301s (hazardous)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How effective are 301 redirects in passing page rank?
I have a blog which is ranking well for certain terms, and would like to repurpose it to better explain these terms it is ranking for, including updating the url to the new term the blog will be about. The plan being to 301 redirect the old url to new. In the past, I've done this with other pages, and have actually lost much of the rankings that I had earned on the original URL. What is your take on this? Maybe repurpose blog, but maintain original URL just to be on the safe side? Thanks
Technical SEO | | CitimarineMoz0 -
301 redirect from dynamic url to static page
Hi, i want to redirect from this old link http://www.g-store.gr/product_info.php?products_id=1735/ to this one https://www.g-store.gr/golf-toualetas.html I have done several attempts but with no result. I anyone can help i will appreciate. My website runs in an Apache server with cpanel. Thank you
Technical SEO | | alstam0 -
Search Console Indexed Page Count vs Site:Search Operator page count
We launched a new site and Google Search Console is showing 39 pages have been indexed. When I perform a Site:myurl.com search I see over 100 pages that appear to be indexed. Which is correct and why is there a discrepancy? Also, Search Console Page Index count started at 39 pages on 5/21 and has not increased even though we have hundreds of pages to index. But I do see more results each week from Site:psglearning.com My site is https://wwww.psglearning.com
Technical SEO | | pdowling0 -
Moving site from html to Wordpress site: Should I port all old pages and redirect?
Any help would be appreciated. I am porting an old legacy .html site, which has about 500,000 visitors/month and over 10,000 pages to a new custom Wordpress site with a responsive design (long overdue, of course) that has been written and only needs a few finishing touches, and which includes many database features to generate new pages that did not previously exist. My questions are: Should I bother to port over older pages that are "thin" and have no incoming links, such that reworking them would take time away from the need to port quickly? I will be restructuring the legacy URLs to be lean and clean, so 301 redirects will be necessary. I know that there will be link juice loss, but how long does it usually take for the redirects to "take hold?" I will be moving to https at the same time to avoid yet another porting issue. Many thanks for any advice and opinions as I embark on this massive data entry project.
Technical SEO | | gheh20130 -
Changing a domain name, pages redirection
when changing a domain name, should we redirect all the pages to their new pages or only the indexed pages? Thanks
Technical SEO | | bigrat950 -
To 301 redirect or not to 301 redirect? duplicate content problem www.domain.com and www.domain.com/en/
Hello, If your website is getting flagged for duplicate content from your main domain www.domain.com and your multilingual english domain www.domain.com/en/ is it wise to 301 redirect the english multilingual website to the main site? Please advise. We've recently installed the joomish component to one of our joomla websites in an effort to streamline a spanish translation of the website. The translation was a success and the new spanish webpages were indexed but unfortunately one of the web developers enabled the english part of the component and some english webpages were also indexed under the multilingual english domain www.domain.com/en/ and that flagged us for duplicate content. I added a 301 redirect to redirect all visitors from the www.domain/en/ webpages to the main www.domain.com/ webpages. But is that the proper way of handling this problem? Please advise.
Technical SEO | | Chris-CA0 -
Robots.txt for subdomain
Hi there Mozzers! I have a subdomain with duplicate content and I'd like to remove these pages from the mighty Google index. The problem is: the website is build in Drupal and this subdomain does not have it's own robots.txt. So I want to ask you how to disallow and noindex this subdomain. Is it possible to add this to the root robots.txt: User-agent: *
Technical SEO | | Partouter
Disallow: /subdomain.root.nl/ User-agent: Googlebot
Noindex: /subdomain.root.nl/ Thank you in advance! Partouter0 -
Redirect or not to redirect
We are rebuilding a website and try to get rid of errors. The content remains exactly the same but we correct the code and make it load faster. The site has quite many backlinks and I can't decide whether to remove .html endings from the urls and 301 redirect to the new ones or leave them with the older ending. If I remove the endings how much of the link juice will be passed? Anyone any idea?
Technical SEO | | sesertin0