Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Technical SEO
    4. Multiple robots.txt files on server

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Multiple robots.txt files on server

    Technical SEO
    5
    7
    3720
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • mjukhud
      mjukhud last edited by

      Hi!

      I have previously hired a developer to put up my site and noticed afterwards that he did not know much about SEO. This lead me to starting to learn myself and applying some changes step by step.

      One of the things I am currently doing is inserting sitemap reference in robots.txt file (which was not there before). But just now when I wanted to upload the file via FTP to my server I found multiple ones - in different sizes - and I dont know what to do with them? Can I remove them? I have downloaded and opened them and they seem to be 2 textfiles and 2 dupplicates. Names:

      robots.txt (original dupplicate)
      robots.txt-Original (original)
      robots.txt-NEW (other content)
      robots.txt-Working (other content dupplicate)

      Would really appreciate help and expertise suggestions. Thanks!

      1 Reply Last reply Reply Quote 0
      • peakdistrictseo
        peakdistrictseo last edited by

        So what's the best policy if a site uses an e-commerce platform like Magento, which has a robots file, but also has a Wordpress blog installed to another folder. eg: /blog and uses a plugin like YOAST which generated a robots file of the Wordpress installation.

        Then you have 2 robots files, is this detrimental or no big deal?

        1 Reply Last reply Reply Quote 0
        • mjukhud
          mjukhud @seoman10 last edited by

          Thanks very much for the help!

          1 Reply Last reply Reply Quote 0
          • mjukhud
            mjukhud last edited by

            Thanks very much for the help!

            1 Reply Last reply Reply Quote 0
            • seoman10
              seoman10 last edited by

              Keep a backup and remove them.

              Search engines are only going to look at the file which is exactly called robots.txt variations of file name will be ignored.

              Do make sure the entries are correct in the main one though, you don't want Google crawling admin pages or other confidential areas of the site.

              mjukhud 1 Reply Last reply Reply Quote 1
              • mjukhud
                mjukhud @Mustansar last edited by

                Hi, thanks for the answer and help!

                Well, I only have one domain that has a webpage and no subdomains active (no blog-subdomain or similar) - so how can I configure that to the situation? Can I just remove all and upload the one I want, maybe?

                1 Reply Last reply Reply Quote 0
                • Mustansar
                  Mustansar last edited by

                  That's a good question, EMS.  The robots.txt protocol can get kind of 
                  confusing when you think about it too long, and it sounds like you've 
                  thought about this a bit.  However, in this case, it might help to 
                  look at robots.txt from the perspective of the spider.

                  When a spider finds a URL, it takes the whole domain name (everything 
                  between 'http://' and the next '/'), then sticks a '/robots.txt' on 
                  the end of it and looks for that file.  If that file exists, then the 
                  spider should read it to see where it is allowed to crawl.

                  In your case, Googlebot, or any other spider, should try to access 
                  three URLs: domainA.com/robots.txt, domainB.domainA.com/robots.txt, 
                  and domainB.com/robots.txt.  The rules in each are treated as 
                  separate, so disallowing robots from domainA.com/ should result in 
                  domainA.com/ being removed from search results while 
                  domainB.domainA.com/ remains unaffected, which does not sound like not 
                  something you want.

                  The problem you might have with the setup you have described is this-- 
                  in order to keep domainB.domainA.com out of the results, you would 
                  need to have domainB.domainA.com/robots.txt exclude robots, while 
                  domainB.com/robots.txt welcomes them.  This means that you would need 
                  to have a way to make domainB.domainA.com/ and domainB.com/ serve 
                  different information, and judging from what you've described, you 
                  have not set up your server to do so yet.

                  Of course, it is always possible that I have assumed to much about 
                  your situation, so it is a good idea to use Google's robots.txt 
                  analysis tool (see http://www.google.com/support/webmasters/bin/topic.py?topic=8475
                  ) to see if your robots.txt files already produce the results you 
                  want.

                  If using robots.txt files doesn't solve the problem, and assuming that 
                  you want to continue hosting all of your content on domainA.com, one 
                  strategy you really should look into would be setting up a 301 
                  redirect from the pages on domainB.domainA.com/ to domainB.com/ .  If 
                  you need more advice on how to do this with your server software, your 
                  hosting company's tech support would definitely be the best place to 
                  start, but this group is here to help if more isues arise. 🙂

                  Hope that helps!

                  mjukhud 1 Reply Last reply Reply Quote 0
                  • 1 / 1
                  • First post
                    Last post

                  Got a burning SEO question?

                  Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                  Start my free trial


                  Browse Questions

                  Explore more categories

                  • Moz Tools

                    Chat with the community about the Moz tools.

                  • SEO Tactics

                    Discuss the SEO process with fellow marketers

                  • Community

                    Discuss industry events, jobs, and news!

                  • Digital Marketing

                    Chat about tactics outside of SEO

                  • Research & Trends

                    Dive into research and trends in the search industry.

                  • Support

                    Connect on product support and feature requests.

                  • See all categories

                  Related Questions

                  • AndyKubrin

                    Robots.txt allows wp-admin/admin-ajax.php

                    Hello, Mozzers!
                    I noticed something peculiar in the robots.txt used by one of my clients: Allow: /wp-admin/admin-ajax.php What would be the purpose of allowing a search engine to crawl this file?
                    Is it OK? Should I do something about it?
                    Everything else on /wp-admin/ is disallowed.
                    Thanks in advance for your help.
                    -AK:

                    Technical SEO | | AndyKubrin
                    2
                  • Leaf-a-mark

                    Redirect multiple domains to 1 domain or not?

                    Hi there, I have client who has multiple domains that already have some PA and DA. Problem is that most websites have the same content and rank better on different keywords.
                    I want to redirect all the websites to 1 domain because it’s easier to manage and it removes any duplicate content. Question is if I redirect domain x to domain y do the rankings of domain x increase on domain y? Or is it better to keep domain x separately to generate more referral traffic to domain y? Thanks in advance! Cheers

                    Technical SEO | | Leaf-a-mark
                    0
                  • Webmaster123

                    I accidentally blocked Google with Robots.txt. What next?

                    Last week I uploaded my site and forgot to remove the robots.txt file with this text: User-agent: * Disallow: / I dropped from page 11 on my main keywords to past page 50. I caught it 2-3 days later and have now fixed it. I re-imported my site map with Webmaster Tools and I also did a Fetch as Google through Webmaster Tools. I tweeted out my URL to hopefully get Google to crawl it faster too. Webmaster Tools no longer says that the site is experiencing outages, but when I look at my blocked URLs it still says 249 are blocked. That's actually gone up since I made the fix. In the Google search results, it still no longer has my page title and the description still says "A description for this result is not available because of this site's robots.txt – learn more." How will this affect me long-term? When will I recover my rankings? Is there anything else I can do? Thanks for your input! www.decalsforthewall.com

                    Technical SEO | | Webmaster123
                    0
                  • ProjectLabs

                    Determining When to Break a Page Into Multiple Pages?

                    Suppose you have a page on your site that is a couple thousand words long. How would you determine when to split the page into two and are there any SEO advantages to doing this like being more focused on a specific topic. I noticed the Beginner's Guide to SEO is split into several pages, although it would concentrate the link juice if it was all on one page. Suppose you have a lot of comments. Is it better to move comments to a second page at a certain point? Sometimes the comments are not super focused on the topic of the page compared to the main text.

                    Technical SEO | | ProjectLabs
                    1
                  • BistosAmerica

                    Oh no googlebot can not access my robots.txt file

                    I just receive a n error message from google webmaster Wonder it was something to do with Yoast plugin. Could somebody help me with troubleshooting this? Here's original message Over the last 24 hours, Googlebot encountered 189 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%. Recommended action If the site error rate is 100%: Using a web browser, attempt to access http://www.soobumimphotography.com//robots.txt. If you are able to access it from your browser, then your site may be configured to deny access to googlebot. Check the configuration of your firewall and site to ensure that you are not denying access to googlebot. If your robots.txt is a static page, verify that your web service has proper permissions to access the file. If your robots.txt is dynamically generated, verify that the scripts that generate the robots.txt are properly configured and have permission to run. Check the logs for your website to see if your scripts are failing, and if so attempt to diagnose the cause of the failure. If the site error rate is less than 100%: Using Webmaster Tools, find a day with a high error rate and examine the logs for your web server for that day. Look for errors accessing robots.txt in the logs for that day and fix the causes of those errors. The most likely explanation is that your site is overloaded. Contact your hosting provider and discuss reconfiguring your web server or adding more resources to your website. After you think you've fixed the problem, use Fetch as Google to fetch http://www.soobumimphotography.com//robots.txt to verify that Googlebot can properly access your site.

                    Technical SEO | | BistosAmerica
                    0
                  • wcksmith

                    Keywords in file names vs folder names

                    We understand the value of a keyword phrase included in the URL.  Is there more value to having that phrase in the folder name of the URL or the file name or does it matter? Example: http://www.biztoolsone.com/website-design.php or http://www.biztoolsone.com/website-design/ Which is best? Thanks,  Wick Smith

                    Technical SEO | | wcksmith
                    0
                  • Grenadi

                    How to 301 multiple domain names to a single domain

                    Hey, I tried to find and answer to this seemingly simple question, but no luck. So, I have one domain name with a website attached to it. I also registered all the other domain names that are similar to it or have different extensions - I want to redirect all the other domain names to my one main domain name without getting penalised by the big G. It looks like this: www.mainsite.com - this is my main domain I also have www.mainsite.com.au, www.mainsite.org, and www.mainsite.org.au which I all want to just redirect to www.mainsite.com I have been told that the best way to do this is a 301 redirect, but to do that you need to make a CNAME for all the other domains that points to www.mainsite.com. My problem is that I cannot seem to create a CNAME record for http://mainsite.com - I have it working for http://www.mainsite.com but not the non www record. What should I be doing differently? Is it just my DNS provider is useless? Thanks, Anthony

                    Technical SEO | | Grenadi
                    0
                  • JordanJudson

                    Should I set up a disallow in the robots.txt for catalog search results?

                    When the crawl diagnostics came back for my site its showing around 3,000 pages of duplicate content. Almost all of them are of the catalog search results page. I also did a site search on Google and they have most of the results pages in their index too. I think I should just disallow the bots in the /catalogsearch/ sub folder, but I'm not sure if this will have any negative effect?

                    Technical SEO | | JordanJudson
                    0

                  Get started with Moz Pro!

                  Unlock the power of advanced SEO tools and data-driven insights.

                  Start my free trial
                  Products
                  • Moz Pro
                  • Moz Local
                  • Moz API
                  • Moz Data
                  • STAT
                  • Product Updates
                  Moz Solutions
                  • SMB Solutions
                  • Agency Solutions
                  • Enterprise Solutions
                  Free SEO Tools
                  • Domain Authority Checker
                  • Link Explorer
                  • Keyword Explorer
                  • Competitive Research
                  • Brand Authority Checker
                  • Local Citation Checker
                  • MozBar Extension
                  • MozCast
                  Resources
                  • Blog
                  • SEO Learning Center
                  • Help Hub
                  • Beginner's Guide to SEO
                  • How-to Guides
                  • Moz Academy
                  • API Docs
                  About Moz
                  • About
                  • Team
                  • Careers
                  • Contact
                  Why Moz
                  • Case Studies
                  • Testimonials
                  Get Involved
                  • Become an Affiliate
                  • MozCon
                  • Webinars
                  • Practical Marketer Series
                  • MozPod
                  Connect with us

                  Contact the Help team

                  Join our newsletter
                  Moz logo
                  © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                  • Accessibility
                  • Terms of Use
                  • Privacy

                  Looks like your connection to Moz was lost, please wait while we try to reconnect.