Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      What is your Brand Authority?
      Moz

      What is your Brand Authority?

      Check yours now
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • SEO Q&A

        Insights & discussions from an SEO community of 500,000+.

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Technical SEO
    4. Removing robots.txt on WordPress site problem

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Removing robots.txt on WordPress site problem

    Technical SEO
    7
    21
    41241
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • Wallander
      Wallander last edited by

      Hi..am a little confused since I ticked the box in WordPress to allow search engines to now crawl my site (previously asked for them not to) but Google webmaster tools is telling me I still have robots.txt blocking them so am unable to submit the sitemap.

      Checked source code and the robots instruction has gone so a little lost. Any ideas please?

      1 Reply Last reply Reply Quote 0
      • nextlevelweb
        nextlevelweb last edited by

        Hi,

        I edited the robots.txt file for my website http://debtfreefrombankruptcy.com yesterday to allow search engines to crawl my site. However, Google isn't recognizing the new file and is still saying that my sitemap is blocked from search. Here is a link to the file itself:

        http://www.debtfreefrombankruptcy.com/robots.txt

        The Blocked URLs tester said that the file allows Google to crawl the site, but in actuality it still isn't recognizing the new file. Any advice would be appreciated. Thanks!

        1 Reply Last reply Reply Quote 0
        • CMcCrone
          CMcCrone last edited by

          I can help you out as this issue DROVE ME NUTS.

          1. I didnt have a Robots.txt (yet)

          2. I had Yoast installed

          3. Im pretty sure it created a Robots.txt even though it doesnt exist in my root (.com/here)

          4. My Google webmaster tools shows this

          User-agent: Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category//* Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /? Disallow: /?Allow: /wp-content/uploadsAllow: /assets Create a Robots.txt

          1. login to wordpress 2. Click SEO in your side toolbar (Yoast WordPress Plugin settings) 3. Go to edit files under SEO (in the side toolbar)

          And now you have the option to edit your Robots.txt file.

          1 Reply Last reply Reply Quote 0
          • evolvingSEO
            evolvingSEO @Wallander last edited by

            Hi Sophia

            I just checked and see your homepage indexed in google.co.uk with a cache date of April 26th. You should be all set!

            -Dan

            1 Reply Last reply Reply Quote 0
            • Wallander
              Wallander last edited by

              Quick update - by amending the robots text file and switching sitemap plugin over to Yoast I finally got the sitemap to index without robots.txt warnings although the Home page of site was not indexed, 'oh dear'.  5 out of the 7 pages in the sitemap were indexed by Google so It's a start but some more investigating to be done on my side.

              evolvingSEO 1 Reply Last reply Reply Quote 0
              • Wallander
                Wallander @Wallander last edited by

                Dan,

                Cant thank you enough! The sitemap request is still pending in Google - maybe I sent too many requests But it's time to sit back and wait for the good news hopefully. Thanks again.

                1 Reply Last reply Reply Quote 0
                • evolvingSEO
                  evolvingSEO @Wallander last edited by

                  Hi Sofia

                  I just ran the same validator on your sitemap and it went through fine - see screenshot

                  I intended to mean that you should just be sure Google Webmaster Tools accepts the sitemap as valid - if so, there's no need to run through a 3rd party validator. Apologies if I didn't state it clearly!

                  Let me know, but from what I can see it looks good!

                  -Dan

                  EDIT - Looking more closely, it looks like your ran the homepage through the validator - you would actually enter the sitemap address its self in the validator - http://containerforsale.co.uk/sitemap.xml

                  1 Reply Last reply Reply Quote 0
                  • Wallander
                    Wallander @Wallander last edited by

                    Hi Dan,

                    I followed the above advice and switched to the Yoast generated sitemap but after testing on http://www.xml-sitemaps.com/validate-xml-sitemap.html I got the following result - no idea what it means but it looks nasty...

                    Schema validating with XSV 3.1-1 of 2007/12/11 16:20:05Schema validator crashed

                    The maintainers of XSV will be notified, you don't need to
                    send mail about this unless you have extra information to provide.
                    If there are Schema errors reported below, try correcting
                    them and re-running the validation.Target: http://containerforsale.co.uk
                       (Real name: http://containerforsale.co.uk
                        Server: Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4)The target was not assessedLow-level XML well-formedness and/or validity processing output
                    Warning: Undefined entity raquo
                     in unnamed entity at line 16 char 83 of http://containerforsale.co.uk
                    Warning: Undefined entity nbsp
                     in unnamed entity at line 160 char 10 of http://containerforsale.co.uk
                    Error: Expected ; after entity name, but got =
                     in unnamed entity at line 274 char 631 of http://containerforsale.co.u

                    1 Reply Last reply Reply Quote 0
                    • evolvingSEO
                      evolvingSEO @Wallander last edited by

                      Sofia

                      You are using Yoast SEO plugin for WordPress, so use the XML sitemap within Yoast. You don't need a separate plugin for the XML sitemap. And yes, within Yoast turn the sitemap on.

                      Hope that helps!

                      -Dan

                      1 Reply Last reply Reply Quote 0
                      • Wallander
                        Wallander @evolvingSEO last edited by

                        Indeed, thanks everyone - it's really appreciated!

                        I have updated the robots.txt as indicated and re submitted site map but looks like Google still has problems with my site since the error warning for robots is there after the processing is done.

                        Quick question - I am using a plugin called Google XML Sitemaps which has the following tick box option.

                        'Add sitemap URL to the virtual robots.txt file'.
                        The virtual robots.txt generated by WordPress is used. A real robots.txt file must NOT exist in the blog directory!'

                        Should this box be ticked or un-ticked please? Fyi I currently don't have the box ticked.

                        evolvingSEO Wallander 4 Replies Last reply Reply Quote 0
                        • evolvingSEO
                          evolvingSEO last edited by

                          Thanks guys for all the responses and helping!

                          Three Things to try

                          1.Fix Robots.txt

                          Sofia - I just checked your robots.txt now and it reads;

                          User-agent: *
                          
                          Disallow: Sitemap: http://containerforsale.co.uk/sitemap.xml.gz
                          
                          • with the sitemap on the same line as disallow - I'd check on that and make sure its on a separate line.
                          • ALSO, you don't need the .gz on the sitemap file just sitemap.xml

                          2. Re-submit Sitemap

                          • RESUBMIT your sitemap to webmaster tools and make sure its valid.

                          3. Submit URL to Webmaster Tools (only last resort)

                          this is only last case scenario, shouldn't have to do this on the homepage if everything is correct.

                          • go to fetch as googlebot ->run the fetch ->then submit URL
                          • do this for the homepage
                          • see article on google blog for reference

                          Let us know if you're all set, thanks!

                          -Dan

                          Wallander 1 Reply Last reply Reply Quote 0
                          • Wallander
                            Wallander @Copstead last edited by

                            Ok thanks Brent, I changed to

                            User-agent: *

                            Disallow:

                            Sitemap: http://containerforsale.co.uk/sitemap.xml.gz

                            Guess I will just have to wait for Google to refresh now...

                            1 Reply Last reply Reply Quote 0
                            • Copstead
                              Copstead @Copstead last edited by

                              yes, the urls being blocked are includes from your Wordpress program.

                              1 Reply Last reply Reply Quote 0
                              • Wallander
                                Wallander @Copstead last edited by

                                Thanks for the heads up.

                                The warning just says 7 Url''s blocked by robots.txt. - have seen this issue posted on the WordPress boards by others but no real insight into solutions.

                                Perhaps I should try your idea of

                                Change the robots.txt file to this:

                                User-agent *

                                Disallow:

                                Sitemap: http://containerforsale.co.uk/sitemap.xml.gz

                                1 Reply Last reply Reply Quote 0
                                • Copstead
                                  Copstead @Copstead last edited by

                                  Well there is a robots.txt file.  You can view it here: http://containerforsale.co.uk/robots.txt

                                  What warnings are you getting in your sitemap submission area?  It appears to look alright: http://containerforsale.co.uk/sitemap.xml  But I tried to validate it and got a 504 Gateway Time-out error. http://www.xml-sitemaps.com/index.php?op=validate-xml-sitemap&go=1&sitemapurl=http%3A%2F%2Fcontainerforsale.co.uk%2Fsitemap.xml&submit=Validate

                                  1 Reply Last reply Reply Quote 0
                                  • Wallander
                                    Wallander @Copstead last edited by

                                    Its weird, the front page warning on Google webmaster for robots has disappeared now, but still got the warnings in the sitemap submission area. My host suggests I just wait a bit longer for Google to update because he said same as you - that there doesn't seem to be any robot.txt file.

                                    1 Reply Last reply Reply Quote 0
                                    • Copstead
                                      Copstead @Wallander last edited by

                                      Doesn't appear to be blocked, so maybe it has something to do with your /wp-includes/ directory.

                                      Change the robots.txt file to this:

                                      User-agent *

                                      Disallow:

                                      Sitemap: http://containerforsale.co.uk/sitemap.xml.gz

                                      Wallander Copstead 5 Replies Last reply Reply Quote 0
                                      • Wallander
                                        Wallander last edited by

                                        Hey Guys,

                                        Thanks for your replies...the domain is http://containerforsale.co.uk ,My host told me to look in the Public HTML file folder for the robots.txt file and just delete it but can't see it in there?

                                        My host said he found a tester site and it doesn't report any issues:

                                        http://www.searchenginepromotionhelp.com/m/robots-text-tester/robots-checker.php

                                        This is the display I get from http://containerforsale.co.uk/robots.txt

                                        User-agent *

                                        Disallow: /wp-admin/
                                        Disallow: /wp-includes/

                                        Sitemap: http://containerforsale.co.uk/sitemap.xml.gz

                                        Copstead 1 Reply Last reply Reply Quote 0
                                        • Anthony_NorthSEO
                                          Anthony_NorthSEO last edited by

                                          Hi Sofia,

                                          Two things you need to consider when troubleshooting this:

                                          The actual robots.txt file (located in the root directory of your site) and the meta-robots tags in the section of your HTML. When you say you checked the source code and the robots instructions were missing, I think you were talking about the meta-robots tags in the actual HTML of your site.

                                          Webmaster Tools is probably referring to the actual robots.txt file in your domain's root path, which would differ entirely and not be visible by checking the HTML on your site. Like Nakul and Brent said, if you'll let us know your site's URL and paste the content of your robots.txt file here, I'm sure one of us can help you resolve the problem fairly quickly.

                                          Thanks!

                                          Anthony

                                          1 Reply Last reply Reply Quote 0
                                          • Copstead
                                            Copstead last edited by

                                            copy whatever you have in your robots.txt file here and we will tell you the issue.

                                            SEOmoz has a great article about Robots.txt files here: http://www.seomoz.org/learn-seo/robotstxt

                                            1 Reply Last reply Reply Quote 1
                                            • NakulGoyal
                                              NakulGoyal last edited by

                                              The robots.txt would probably not be a part of the Wordpress Configuration. Allow indexing is controlled via Meta Data by the Wordpress Architecture.

                                              I would look for something like this in yourdomain.com/robots.txt

                                              disallow /

                                              or something like that. If that does not help, PM me your site URL and I would be glad to look it up for you.

                                              1 Reply Last reply Reply Quote 0
                                              • 1 / 1
                                              • First post
                                                Last post

                                              Got a burning SEO question?

                                              Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                                              Start my free trial


                                              Browse Questions

                                              Explore more categories

                                              • Moz Tools

                                                Chat with the community about the Moz tools.

                                              • SEO Tactics

                                                Discuss the SEO process with fellow marketers

                                              • Community

                                                Discuss industry events, jobs, and news!

                                              • Digital Marketing

                                                Chat about tactics outside of SEO

                                              • Research & Trends

                                                Dive into research and trends in the search industry.

                                              • Support

                                                Connect on product support and feature requests.

                                              • See all categories

                                              Related Questions

                                              • Smileworks_Liverpool

                                                What's the best way for users to upload their images to my wordpress site to promote UGC

                                                I have looked at lots of different plugins and wanted a recommendation for an easy way for patients of ours to upload pictures of them out partying and having fun and looking beautiful so future users can see the final results instead of sometimes gory or difficult to understand before and after images. I'd like to give them the opportunity to write captions (like facebook or insta posts and would offer them incentives to do so. I don't want it to be too complicated for them or have too many steps or barriers but I do want it to look nice and slick and modern. Also do you think this would have a positive impact on SEO? I was also thinking of a Q&A app where dentists could get Q&A emails and respond - i've been doing AMA sessions and they've been really successful and I would like to bring it into out site and make it native. Thanks in advance 🙂

                                                Technical SEO | | Smileworks_Liverpool
                                                1
                                              • MarketHubb

                                                Help Setting Up 301 Redirects from Coldfusion Site to Wordpress Site.

                                                I have created a new website and need to redirect all of the previous pages to the new one. The old website was built in coldfusion and the new site is built in wordpress. One of the pages I'm trying to redirect is www.norriseal.com/products.cfm to http://norrisealwellmark.com/products/. This is what I have in my .htaccess file <ifmodule mod_rewrite.c="">Options +FollowSymlinks
                                                RewriteEngine On
                                                RewriteBase /
                                                Redirect 301 /products.cfm http://norrisealwellmark.com/products/</ifmodule> The result of this redirect is http://norrisealwellmark.com/products.cfm How do I prevent the .cfm from appending to the destination URL?

                                                Technical SEO | | MarketHubb
                                                1
                                              • mjukhud

                                                Multiple robots.txt files on server

                                                Hi! I have previously hired a developer to put up my site and noticed afterwards that he did not know much about SEO. This lead me to starting to learn myself and applying some changes step by step. One of the things I am currently doing is inserting sitemap reference in robots.txt file (which was not there before). But just now when I wanted to upload the file via FTP to my server I found multiple ones - in different sizes - and I dont know what to do with them? Can I remove them? I have downloaded and opened them and they seem to be 2 textfiles and 2 dupplicates. Names: robots.txt (original dupplicate)
                                                robots.txt-Original (original)
                                                robots.txt-NEW (other content)
                                                robots.txt-Working (other content dupplicate) Would really appreciate help and expertise suggestions. Thanks!

                                                Technical SEO | | mjukhud
                                                0
                                              • QuantumWeb62

                                                Removing site subdomains from Google search

                                                Hi everyone, I hope you are having a good week? My website has several subdomains that I had shut down some time back and pages on these subdomains are still appearing in the Google search result pages. I want all the URLs from these subdomains to stop appearing in the Google search result pages and I was hoping to see if anyone can help me with this. The subdomains are no longer under my control as I don't have web hosting for these sites (so these subdomain sites just show a default hosting server page). Because of this, I cannot verify these in search console and submit a url/site removal request to Google. In total, there are about 70 pages from these subdomains showing up in Google at the moment and I'm concerned in case these pages have any negative impacts on my SEO. Thanks for taking the time to read my post.

                                                Technical SEO | | QuantumWeb62
                                                0
                                              • emasaa

                                                Removing Media from Wordpress

                                                I've run the seomoz on page report and found an interesting issue. I'm using wordpress and it seems that every picture I add to my articles seem to be added as separate pages to the site. I'm having to go to each and every picture and creating a meta tag and description to it. I still get duplicate content issues with the same. On my Disqus system, I get the same pictures added just as a page or article would look like. What can I do to avoid this?

                                                Technical SEO | | emasaa
                                                0
                                              • HMK-NL

                                                No indexing url including query string with Robots txt

                                                Dear all, how can I block url/pages with query strings like page.html?dir=asc&order=name with robots txt? Thanks!

                                                Technical SEO | | HMK-NL
                                                0
                                              • irvingw

                                                Allow or Disallow First in Robots.txt

                                                If I want to override a Disallow directive in robots.txt with an Allow command, do I have the Allow command before or after the Disallow command? example: Allow: /models/ford///page* Disallow: /models////page

                                                Technical SEO | | irvingw
                                                0
                                              • travelocitysearch

                                                Google.ca is showing our US site instead of our Canada Site

                                                When our Canadian users who search on google.ca for our brand (e.g. Travelocity, Travelocity hotels, etc.), the first few results our from our US site (travelocity.com) rather than our Canadian site (travelocity.ca). In Google Webmaster Tools, we've adjusted the geotargeting settings to focus on the appropriate locale, but the wrong country TLD is still coming up at the top via google.ca. What's the best way to ensure our Canadian site comes up instead of the US site on google.ca? Thanks, Tory Smith
                                                Travelocity

                                                Technical SEO | | travelocitysearch
                                                0

                                              Get started with Moz Pro!

                                              Unlock the power of advanced SEO tools and data-driven insights.

                                              Start my free trial
                                              Products
                                              • Moz Pro
                                              • Moz Local
                                              • Moz API
                                              • Moz Data
                                              • STAT
                                              • Product Updates
                                              Moz Solutions
                                              • SMB Solutions
                                              • Agency Solutions
                                              • Enterprise Solutions
                                              Free SEO Tools
                                              • Domain Authority Checker
                                              • Link Explorer
                                              • Keyword Explorer
                                              • Competitive Research
                                              • Brand Authority Checker
                                              • MozBar Extension
                                              • MozCast
                                              Resources
                                              • Blog
                                              • SEO Learning Center
                                              • Help Hub
                                              • Beginner's Guide to SEO
                                              • How-to Guides
                                              • Moz Academy
                                              • API Docs
                                              About Moz
                                              • About
                                              • Team
                                              • Careers
                                              • Contact
                                              Why Moz
                                              • Case Studies
                                              • Testimonials
                                              Get Involved
                                              • Become an Affiliate
                                              • MozCon
                                              • Webinars
                                              • Practical Marketer Series
                                              • MozPod
                                              Connect with us

                                              Contact the Help team

                                              Join our newsletter

                                              Access all your tools in one place. Whether you're tracking progress or analyzing data, everything you need is at your fingertips.

                                              Moz logo
                                              © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                                              • Accessibility
                                              • Terms of Use
                                              • Privacy

                                              Looks like your connection to Moz was lost, please wait while we try to reconnect.