Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • MozCon
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      Track AI Overviews in Keyword Research
      Moz Pro

      Track AI Overviews in Keyword Research

      Try it free!
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Technical SEO
    4. Robots txt. in page with 301 redirect

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Robots txt. in page with 301 redirect

    Technical SEO
    2
    6
    4358
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • Kilgray
      Kilgray last edited by

      We currently have a a series of help pages that we would like to disallow from our robots txt.

      The thing is that these help pages are located in our old website, which now has a 301 redirect to current site.

      Which is the proper way to go around?

      1- Add the pages we want to disallow to the robots.txt of the new website?

      2- Break the redirect momentarily and add the pages to the robots.txt of the old one?

      Thanks

      1 Reply Last reply Reply Quote 0
      • LauraSultan
        LauraSultan @Kilgray last edited by

        In that case, you'd need to add the robots meta tag at the page level before the tag.

        or

        1 Reply Last reply Reply Quote 1
        • Kilgray
          Kilgray @LauraSultan last edited by

          Hey, for some time we will keep the files in the old domain. Should we break the redirect and insert the disallows to the robot.txt of the old site?

          LauraSultan 1 Reply Last reply Reply Quote 0
          • LauraSultan
            LauraSultan @Kilgray last edited by

            So, the problem is that the robots.txt file can't be accessed because of the 301 redirect to the new domain?

            Do you plan to keep the help files on the old domain, or will they be removed completely?

            Kilgray 1 Reply Last reply Reply Quote 0
            • Kilgray
              Kilgray @LauraSultan last edited by

              Hi Laura,

              Thanks for your reply. I don't want to disallow the URLs these pages are being redirected to. Actually these URLs are in the old version but still can be accessed. So to put it simply, this is my case:

              1- This was our current website: www.kilgray.com (With a 301 redirect)

              2- This is our new website: www.memoq.com

              3- I would like to disallow the following links on the old website that are still visible (haven't been redirected):

              http://kilgray.com/memoq/2015-100/help-en/index.html

              http://kilgray.com/memoq/2014/help-en/

              http://kilgray.com/memoq/2013R2/help-full-en/

              http://kilgray.com/memoq/60/help-en/

              LauraSultan 1 Reply Last reply Reply Quote 0
              • LauraSultan
                LauraSultan last edited by

                Do you want to disallow the URLs that these pages are being redirected to? If not, there's no need to add anything to the robots.txt file.

                If you do want to disallow the URLs that these pages are being redirected to, use relative URLs in your robots.txt file. For example, let's say olddomain.com/old-help-page/ is being redirected to newdomain.com/new-help-page/. If that's the case, add the following to your robots.txt file.

                Disallow: /new-help-page/

                There's no need to disallow the specific URLs that are being redirected to something else.  Are you trying to get them removed from Google's index or something?  If so, Google will update their index eventually based on your 301 redirects.

                Kilgray 1 Reply Last reply Reply Quote 0
                • 1 / 1
                • First post
                  Last post

                Got a burning SEO question?

                Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                Start my free trial


                Browse Questions

                Explore more categories

                • Moz Tools

                  Chat with the community about the Moz tools.

                • SEO Tactics

                  Discuss the SEO process with fellow marketers

                • Community

                  Discuss industry events, jobs, and news!

                • Digital Marketing

                  Chat about tactics outside of SEO

                • Research & Trends

                  Dive into research and trends in the search industry.

                • Support

                  Connect on product support and feature requests.

                • See all categories

                Related Questions

                • aliho

                  What is the difference between 301 redirects and backlinks?

                  i have seen some 301 redirects on my site  billsonline, can anyone please explain the difference between backlinks and 301 redirects,  i have read some articles where the writer was stating  that  301 are not good for website.

                  Technical SEO | | aliho
                  0
                • zeepartner

                  Robots.txt on http vs. https

                  We recently changed our domain from http to https. When a user enters any URL on http, there is an global 301 redirect to the same page on https. I cannot find instructions about what to do with robots.txt. Now that https is the canonical version, should I block the http-Version with robots.txt? Strangely, I cannot find a single ressource about this...

                  Technical SEO | | zeepartner
                  0
                • imaginex

                  Should I block Map pages with robots.txt?

                  Hello, I have a website that was started in 1999. On the website I have map pages for each of the offices listed on my site, for which there are about 120. Each of the 120 maps is in a whole separate html page. There is no content in the page other than the map. I know all of the offices love having the map pages so I don't want to remove the pages. So, my question is would these pages with no real content be hurting the rankings of the other pages on our site? Therefore, should I block the pages with my robots.txt? Would I also have to remove these pages (in webmaster tools?) from Google for blocking by robots.txt to really work? I appreciate your feedback, thanks!

                  Technical SEO | | imaginex
                  0
                • praveen439

                  Sudden drop in Rankings after 301 redirect

                  Greetings to Moz Community. Couple of months back, I have redirected my old blog to a new URL with 301 redirect because of spammy links pointed to my old blog. I have transfer all the posts manually, changed the permalink structure and 301 redirected every individual URL. All the ranking were boosted within couple of weeks and regained the traffic. After a month I have observed, the links pointed to old site are showing up in Webmaster Tools for the new domain. I was shocked (no previous experience) and again Disavowed all links. Today, all the positions went down for new domain. My questions are: 1. Did the Disavow tool worked this time with new domain? All the links pointed to old domain were devaluated? Is this the reason for ranking drop? Or 2. 301 Old domain with Unnatural links causes the issue? 3. Removing 301 will help to regain few keyword positions? I'm taking this as a case study. Already removed the 301 redirect. Looking for solid discussion.Thanks.

                  Technical SEO | | praveen439
                  0
                • Natitude

                  Best Practice on 301 Redirect - Images

                  We have two sites that sell the same products. We have decided to retire one of the sites as we'd like to focus on one property. I know best practice is to redirect apples to apples, which in our case is easily done since the sites sold the same thing. www.SiteABC.com/ProductA can be redirected to www.SiteXYZ.com/ProductA. My question is how far does that thinking go regarding images? Each product has a main product page, of course, and then up to 6 images in some cases. Is it necessary to redirect www.SiteABC.com/ProductA-Image1.jpg to www.SiteXYZ.com/ProductA-Image1.jpg? Or can they all be redirected to just the product page?

                  Technical SEO | | Natitude
                  0
                • SDSLaw

                  Questions about the Sandbox and 301 Redirects

                  Does the sandbox still exist?  What if you have a brand new URL and do a 301 redirect from another website because the name of the service business changed? Thanks for any insight and help.

                  Technical SEO | | SDSLaw
                  0
                • TroyW

                  301 Redirect with index.asp

                  I am very new to all of this so forgive the newbie questions I will get better.   Ok so after starting a campaign I see that I have many issues including where some pages are being deemed as duplicate content. 1. The report says the http://lucid8.com has duplicate content on 2 other pages 2. When I look at them it shows that http://lucid8.com/index.asp and http://www.lucid8.com are duplicates. 3. Really these are the exactly the same page because the default page that is opened for www.lucid8.com http://www.lucid8.com etc always opens the index.asp page. 4. Now I read that I should do permanent redirects and how to do this via IIS and I tried to do a redirect from index.asp to www.lucid8.com but that does not work because www.lucid8.com is pointing to index.asp and so we end up in a circle. So the question is how do I get rid of these duplicate page references without causing problems. Thanks

                  Technical SEO | | TroyW
                  0
                • EcomLkwd

                  Changing title tags, do we need 301 redirects

                  I found many duplicate title tags and I'm in the process of changing it  Do I need 301 redirects in place when I switch it? I am only changing the title tag. Also, we are switching over to a new site very soon, I am worried that we might be using too many 301 redirect "hops" because we are doing a lot of optimization as well. (video from matt cutts describing 301 redirects and hops: http://www.youtube.com/watch?v=r1lVPrYoBkA. Does anyone have any experience in doing too many redirect hops that it affected your rankings? Any good ideas to avoid this?

                  Technical SEO | | EcomLkwd
                  0

                Get started with Moz Pro!

                Unlock the power of advanced SEO tools and data-driven insights.

                Start my free trial
                Products
                • Moz Pro
                • Moz Local
                • Moz API
                • Moz Data
                • STAT
                • Product Updates
                Moz Solutions
                • SMB Solutions
                • Agency Solutions
                • Enterprise Solutions
                • Digital Marketers
                Free SEO Tools
                • Domain Authority Checker
                • Link Explorer
                • Keyword Explorer
                • Competitive Research
                • Brand Authority Checker
                • Local Citation Checker
                • MozBar Extension
                • MozCast
                Resources
                • Blog
                • SEO Learning Center
                • Help Hub
                • Beginner's Guide to SEO
                • How-to Guides
                • Moz Academy
                • API Docs
                About Moz
                • About
                • Team
                • Careers
                • Contact
                Why Moz
                • Case Studies
                • Testimonials
                Get Involved
                • Become an Affiliate
                • MozCon
                • Webinars
                • Practical Marketer Series
                • MozPod
                Connect with us

                Contact the Help team

                Join our newsletter
                Moz logo
                © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                • Accessibility
                • Terms of Use
                • Privacy

                Looks like your connection to Moz was lost, please wait while we try to reconnect.