Bloking pages in roborts.txt that are under a redirected subdomain
-
Hi Everyone,
I have a lot of Marketo landing pages that I don't want to show in SERP. Adding the noindex meta tag for each page will be too much, I have thousands of pages.
Blocking it in roborts.txt could have been an option, BUT, the subdomain homepage is redirected to my main domain (with a 302) so I may confuse search engines ( should they follow the redirect or should they block)
marketo.mydomain.com is redirected to www.mydomain.com
disallow: / (I think this will be confusing with the redirect)
I don't have folders, all pages are under the subdomain, so I can't block folders in Robots.txt also
Would anyone had this scenario or any suggestions?
I appreciate your thoughts here.
Thank you
Rachel
-
Thank you so much for you answer!
the home page in the subdomain is redirected but none of the actual pages in the subdomain are, and because there are so many of them, it would be easier to block them in robots.txt, even if there is small change that Google will still index them. But because the home page is redirected, I don't want to confuse Google with a Disallow: /
Could I do Disallow: / and then Allow: /homepage.html
-
Under usual circumstances, Google won't index redirecting addresses which it considers to be a shallow form of 'doorway' page (Google doesn't like to rank those). If I am reading your post right, no pages on the redirected sub-domain can be visited as they all now redirect. Google should start dropping those old URLs from its index automatically. It's important to note that you should be using 301s to eventually de-index the old URLs. If you're using 302s then you're telling Google that the old pages are only being redirected temporarily and they will return (which could mess with indexation)
If I am reading your post wrong and some pages are still live on the old subdomain and are not redirecting, and you want Google to redirect most addresses but some are staying (for whatever reason) and you still want to de-index those specific ones, Meta no-index (sorry) really is your best bet! Robots.txt tells Google not to crawl a page but it doesn't tell Google not to index a URL if external metrics (inbound links) are strong enough. Both are 'directives' and Google is forced to obey neither
I think from what you are saying, I'd just leave the redirects and let Google do its work. Make sure they're 301s, though. You're right that Robots.txt might end up confusing things. Robots.txt can also sometimes 'kill' the SEO authority of a page. If you did that for all the redirecting pages, no equity would flow through your 301s (hazardous)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is my inner pages ranking higher than main page?
Hi everyone, for some reason lately i have discovered that Google is ranking my inner pages higher than the main subfolder page. www.domain.com/subfolder --> Target page to be ranked
Technical SEO | | davidboh
www.domain.com/subfolder/aboutus ---> page that is currently ranking Also in the SERP most of the time, it is showing both links in this manner. www.domain.com/subfolder/aboutus
-----------www.domain.com/subfolder Thanks in advance.1 -
Over 40+ pages have been removed from the indexed and this page has been selected as the google preferred canonical.
Over 40+ pages have been removed from the indexed and this page has been selected as the google preferred canonical. https://studyplaces.com/about-us/ The pages affected by this include: https://studyplaces.com/50-best-college-party-songs-of-all-time-and-why-we-love-them/ https://studyplaces.com/15-best-minors-for-business-majors/ As you can see the content on these pages is totally unrelated to the content on the about-us page. Any ideas why this is happening and how to resolve.
Technical SEO | | pnoddy0 -
Subdomain or subfolder?
Hello, We are working on a new site. The idea of the site is to have an ecommerce shop, but the homepage will be a content page, basically a blog page.
Technical SEO | | pinder325
My developer wants to have the blog (home) page on a subdomain, so blog.example.com, because it will be easier to make a nice content page this way, and the the rest of the site will just be on the root domain (example.com). I'm just worried that this will be bad for our SEO efforts. I've always thought it was better to use a sub folder rather than a subdomain. If we get links to the content on the subdomain, will the link juice flow to the shop, on the root domain? What are your thoughts?0 -
Redirecting a blog
We've acquired another company and want to redirect their soon-to-be-obsolete website to ours. It includes a blog with many blog posts. Should we: only 301 redirect the top level blog URL
Technical SEO | | Caro-O
try redirect individual blogs to blogs of a similar topic on our site (least practical I'm sure)
redirect all their individual posts to our main blog URL Thanks, Caro1 -
Many Pages Being Combined Into One Long Page
Hi All, In talking with my internal developers, UX, and design team there has been a big push to move from a "tabbed" page structure (where as each tab is it's own page) to combining everything into one long page. It looks great from a user experience standpoint, but I'm concerned that we'll decrease in rankings for the tabbed pages that will be going away, even with a 301 in place. I initially recommending#! or pushstate for each "page section" on the long form content. However there are technical limitations with this in our CMS. The next idea I had was to still leave those pages out there and to link to them in the source code, but this approach may get shot down as well. Has anyone else had to solve for this issue? If so, how did you do it?
Technical SEO | | AllyBank1 -
Redirect Without Passing Old Page Properties
Is there a way to redirect one page to another, e.g. test.com/ to test.com/home, without passing link juice or any other associated properties of the latter to the former?
Technical SEO | | NTGproducts0 -
Index page
To the SEO experts, this may well seem a silly question, so I apologies in advance as I try not to ask questions that I probably know the answer for already, but clarity is my goal I have numerous sites ,as standard practice, through the .htaccess I will always set up non www to www, and redirect the index page to www.mysite.com. All straight forward, have never questioned this practice, always been advised its the ebst practice to avoid duplicate content. Now, today, I was looking at a CMS service for a customer for their website, the website is already built and its a static website, so the CMS integration was going to mean a full rewrite of the website. Speaking to a friend on another forum, he told me about a service called simple CMS, had a look, looks perfect for the customer ... Went to set it up on the clients site and here is the problem. For the CMS software to work, it MUST access the index page, because my index page is redirected to www.mysite.com , it wont work as it cant find the index page (obviously) I questioned this with the software company, they inform me that it must access the index page, I have explained that it wont be able to and why (cause I have my index page redirected to avoid duplicate content) To my astonishment, the person there told me that duplicate content is a huge no no with Google (that's not the astonishing part) but its not relevant to the index and non index page of a website. This goes against everything I thought I knew ... The person also reassured me that they have worked within the SEO area for 10 years. As I am a subscriber to SEO MOZ and no one here has anything to gain but offering advice, is this true ? Will it not be an issue for duplicate content to show both a index page and non index page ?, will search engines not view this as duplicate content ? Or is this SEO expert talking bull, which I suspect, but cannot be sure. Any advice would be greatly appreciated, it would make my life a lot easier for the customer to use this CMS software, but I would do it at the risk of tarnishing the work they and I have done on their ranking status Many thanks in advance John
Technical SEO | | Johnny4B0 -
301 redirects
At the moment it's possible to access the home page of my website via two different urls, with and without www. and you've told me that this can be resolved with Canonicalization and a 301 redirect. Do I do this with my web hosting package or in my html pages? If I can't do it with my web host (1&1) then is there an idiot's guide of how to do it yourself? I've also got both the domain vamospaella.co.uk and vamospaella.com. Is it better to have one of these redirecting to the other for UK traffic (at the moment .co.uk redirects to .com) Thanks
Technical SEO | | melissa10