Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Should I add my html sitemap to Robots?

Technical SEO

653

Trazo last edited by

I have already added the .xml to Robots. But should I also add the html version?
1 Reply Last reply
Reply Quote 0
Libra_Photographic last edited by

As said above, great question though.
1 Reply Last reply
Reply Quote 1
Martijn_Scheijbeler last edited by

No, it won't help you at all as it's not a valid extension that they will use. What you can do is add a link to the HTML sitemap from multiple pages on your site so you provide an efficient way for Google to access it and use it to crawl the other pages on your site.
1 Reply Last reply
Reply Quote 4

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Google Search console says 'sitemap is blocked by robots?

Google Search console is telling me "Sitemap contains URLs which are blocked by robots.txt." I don't understand why my sitemap is being blocked? My robots.txt look like this: User-Agent: *
Disallow: Sitemap: http://www.website.com/sitemap_index.xml It's a WordPress site, with Yoast SEO installed. Is anyone else having this issue with Google Search console? Does anyone know how I can fix this issue?
Technical SEO | | Extima-Christian

1
Robots.txt blocking Addon Domains

I have this site as my primary domain: http://www.libertyresourcedirectory.com/ I don't want to give spiders access to the site at all so I tried to do a simple Disallow: / in the robots.txt. As a test I tried to crawl it with Screaming Frog afterwards and it didn't do anything. (Excellent.) However, there's a problem. In GWT, I got an alert that Google couldn't crawl ANY of my sites because of robots.txt issues. Changing the robots.txt on my primary domain, changed it for ALL my addon domains. (Ex. http://ethanglover.biz/ ) From a directory point of view, this makes sense, from a spider point of view, it doesn't. As a solution, I changed the robots.txt file back and added a robots meta tag to the primary domain. (noindex, nofollow). But this doesn't seem to be having any effect. As I understand it, the robots.txt takes priority. How can I separate all this out to allow domains to have different rules? I've tried uploading a separate robots.txt to the addon domain folders, but it's completely ignored. Even going to ethanglover.biz/robots.txt gave me the primary domain version of the file. (SERIOUSLY! I've tested this 100 times in many ways.) Has anyone experienced this? Am I in the twilight zone? Any known fixes? Thanks. Proof I'm not crazy in attached video. robotstxt_addon_domain.mp4
Technical SEO | | eglove

0
Sitemap question

Hello, In your opinion what is better for a root domain and micro-sites using sub-domains?, to have a single sitemap for the root domain including all links to the sub-domains or to have a separate sitemap for each sub-domain? Thanks Arnold
Technical SEO | | arnoldwender

0
How can I find my Webmaster Tools HTML file?

So, totally amateur hour here, but I can't for the life of me find our HTML verification file for webmaster tools. I see nowhere to look at it in Google Webmaster Tools console, I tried a site:, I googled it, all the info out there is about how to verify a site. Ours is verified, but I need the verification file code to sync up with the Google API and no one seems to have it. Any thoughts?
Technical SEO | | healthgrades

0
Using Sitemap Generator - Good/Bad?

Hi all I recently purchased the full licence of XML Sitemap Generator (http://www.xml-sitemaps.com/standalone-google-sitemap-generator.html) but have yet used it. The idea behind this is that I can deploy the package on each large e-commerce website I build and the sitemap will be generated as often as I set it be and the search engines will also be pinged automatically to inform them of the update. No more manual XML sitemap creation for me! Now it sounds great but I do not know enough about pinging search engines with XML sitemap updates on a regular basis and if this is a good or bad thing? Can it have any detrimental effect when the sitemap is changing (potentially) every day with new URLs for products being added to the site? Any thoughts or optinions would be greatly appreciated. Kris
Technical SEO | | yousayjump

0
Sitemap for 170 K webpages

I have 170 K pages on my website which I want to be indexed. I have created a multiple HTML sitemaps (e.g. sitemap1.html, sitemap2.html,...etc) with each sitemap page having 3000 links. Is this right approach or should i switch to xml based sitemaps and that too multiple one. Please suggest.
Technical SEO | | ArtiKalra

0
Sitmap Page - HTML and XML

Hi there I have a domain which has a sitemap in html for regular users and a sitemap in xml for the spiders. I have a warning via seomoz saying that i have too many links on the html version. What do i do here? regards Stef
Technical SEO | | stefanok

0
How do I add meta descriptions to Archives in Wordpress?

My most recent crawl returned a number of 'missing meta description' errors, and when I checked individual URLs, it turned out they were Wordpress Archived pages - for individual months and days (e.g. http:// .../2011/01). What's the best way to go about adding descriptions to these pages, if at all? Or should I have these pages not be indexed? I am using the All in One SEO plugin, so maybe there is an easy fix through this plugin, or it may be the cause of these errors? Any help is appreciated, thanks in advance! **EDIT After looking it up further, I have decided to use noindex for Archives, which should solve my problem right? Or is there a benefit to having those archived pages?
Technical SEO | | NetPicks

0