Wordpress Robots.txt Sitemap submission?
-
Alright, my question comes directly from this article by SEOmoz
http://www.seomoz.org/learn-seo/r...
Yes, I have submitted the sitemap to google, bing's webmaster tools and
and I want to add the location of our site's sitemaps and does it mean that I erase everything in the robots.txt right now and replace it with?
<code>User-agent: * Disallow: Sitemap: http://www.example.com/none-standard-location/sitemap.xml</code>
<code>???</code>
because Wordpress comes with some default disallows like wp-admin, trackback, plugins. I have also read this, but was wondering if this is the correct way to add sitemap on Wordpress Robots.txt. [http://www.seomoz.org/q/removing-...](http://www.seomoz.org/q/removing-robots-txt-on-wordpress-site-problem) I am using Multisite with Yoast plugin so I have more than one sitemap.xml to submit Do I erase everything in Robots.txt and replace it with how SEOmoz recommended? hmm that sounds not right. like
<code>
<code>
<code>User-agent: *
Disallow: </code>Sitemap: http://www.example.com/sitemap_index.xml</code>
<code>``` Sitemap: http://www.example.com/sub/sitemap_index.xml ```</code>
<code>?????????</code> ```</code>
-
Hi!
Definitely don't delete what's in the robots.txt file now. Simply add the sitemap lines at the top of the file, one per line - like this example from the WordPress documentation
Leave everything is intact below it that is already in your robots.txt file. You probably don't have as much "stuff" as this example, so just add your sitemaps at the top and you should be all set.
-Dan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Getting 'Indexed, not submitted in sitemap' for around a third of my site. But these pages ARE in the sitemap we submitted.
As in the title, we have a site with around 40k pages, but around a third of them are showing as "Indexed, not submitted in sitemap" in Google Search Console. We've double-checked the sitemaps we have submitted and the URLs are definitely in the sitemap. Any idea why this might be happening? Example URL with the error: https://www.teacherstoyourhome.co.uk/german-tutor/Egham Sitemap it is located on: https://www.teacherstoyourhome.co.uk/sitemap-subject-locations-surrey.xml
Technical SEO | | TTYH0 -
Robots.txt error
Moz Crawler is not able to access the robots.txt due to server error. Please advice on how to tackle the server error.
Technical SEO | | Shanidel0 -
What is the value of having an HTML sitemap on site?
For years now whenever we recreate a site we always set up both an xml sitemap and an html sitemap page. Stupid question maybe - but what is the value to having an html sitemap on site?
Technical SEO | | Pete40 -
Best XML Generator for Wordpress?
Hi all, Quick question - does anybody have any recommendations for the best XML sitemap plugin for Wordpress? An idea of why you like it would also be helpful. Thank you very much! Mark
Technical SEO | | markadoi840 -
Sitemap Creation
Hi I am looking for the best way to generate an XML sitemap for webmaster tools for my website http://www.cheapfindergames.com. I have come across http://www.xml-sitemaps.com/ but it only allows up to 500 links. Is there a PHP script that any experts could share that would create the XML map that I could upload please? Many Thanks
Technical SEO | | ocelot0 -
Robots.txt question
What is this robots.txt telling the search engines? User-agent: * Disallow: /stats/
Technical SEO | | DenverKelly0 -
Rel Canonical - Wordpress
How do you fix the rel canonical issue on a wordpress site? Is there a quick fix? I have a few notices on my site and am a little confused. Thanks, Jared
Technical SEO | | SaborStyle0 -
Sitemaps - Format Issue
Hi, I have a little issue with a client site whose programmer seems kind of unwilling to change things that he has been doing a long time. So, he has had this dynamic site set up for a few years and active in google webmaster tools and others, but is not happy with the traffic it is getting. When I looked at webmaster tools I see that he has a sitemap registered, but it is /sitemap.php When I said that we should be offering the SE's /sitemap.xml his response is that sitemap.php checks the site every day and generates /sitemap.xml, but there is no /sitemap.xml registered in webmaster tools. My gut is telling me that he should just register /sitemap.xml in webmaster tools, but it is a hard sell 🙂 Anyone have any definitive experience of people doing this before and whether it is an issue? My feeling is that it doesn't need to be rocket science... Any input appreciated, Sha
Technical SEO | | ShaMenz0