Protecting sitemaps - Good idea or humbug?
-
Is there a way to protect your sitemap.xml so that only Google can read it and would it make sense to do this?
-
From a hacker's perspective, the first order of business is going to be gathering information on the target. does a hacker or someone with malicious intent gain something in obtaining access to your sitemap?
Yes, they do, and that is more information on the layout of your site. How common would there actually be something on the sitemap that could critically expose you to compromise on your VPS/Shared hosting? Um, probably super ultra rare.
But yes there was one time that I was doing an audit for a company and the sitemap did point to a directory that was vulnerable to directory browsing. Fishing around in the directory, I was able to obtain a picture of a PayPal MasterCard front and back because some idiot snapped pictures of it and uploaded it onto the site.
So there are benefits to hiding it, it's relatively easy to do, but if your lazy and don't want to, chances are your good.
-
Hi Herb,
Thank you for your feedback. I think you are right. We are dealing with very short lived up-to-date information so it is vital that as few sites as possible have the information we have. For this reason I was considering to "hide" our sitemaps. Some of our competitors do that but probably we need to find some other measures to achieve our goal.
Cheers
Thomas -
Hi Thomas;
You have not specified your web server platform, but assuming it is Apache it would be easy to do with a regular expression in your .htaccess
However, I do not see any valid reason for doing so. Your sitemap should be a refection of your public menu and internal public links. So other than making it easier for search and other spiders to crawl your site, it does not expose any information that is not available by other methods. So, best practices say that you should have an accurate site map, and unless you have a reson for hiding it that you did not mention I would not hide it.
I will tell you those that you should not bother putting areas you do not want crawled in your robots.txt file and any of the bad folks will not respect the request.
Take care,
Herb
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
The W3C Markup Validation Service - Good, Bad or Impartial?
Hi guys, it seems that now days it is almost impossible to achieve 0 (Zero) Errors when testing a site via (The W3C Markup Validation Service - https://validator.w3.org). With analytic codes, pixels and all kind of tracking and social media scripts gunning it seems to be an unachievable task. My questions to you fellow SEO'rs out there are 2: 1. How important and to what degree of effort do you go when you technically review a site and make the decision as to what needs to be fixed and what you shouldn't bother with. 2. How do you argue your corner when explaining to your clients that its impossible to active 100% validation. *As a note i will say that i mostly refer to Wordpress driven sites. would love ot hear your take. Daniel.
Technical SEO | | artdivision0 -
Clarification on indexation of XML sitemaps within Webmaster Tools
Hi Mozzers, I have a large service based website, which seems to be losing pages within Google's index. Whilst working on the site, I noticed that there are a number of xml sitemaps for each of the services. So I submitted them to webmaster tools last Friday (14th) and when I left they were "pending". On returning to the office today, they all appear to have been successfully processed on either the 15th or 17th and I can see the following data: 13/08 - Submitted=0 Indexed=0
Technical SEO | | Silkstream
14/08 - Submitted=606,733 Indexed=122,243
15/08 - Submitted=606,733 Indexed=494,651
16/08 - Submitted=606,733 Indexed=517,527
17/08 - Submitted=606,733 Indexed=517,498 Question 1: The indexed pages on 14th of 122,243 - Is this how many pages were previously indexed? Before Google processed the sitemaps? As they were not marked processed until 15th and 17th? Question 2: The indexed pages are already slipping, I'm working on fixing the site by reducing pages and improving internal structure and content, which I'm hoping will fix the crawling issue. But how often will Google crawl these XML sitemaps? Thanks in advance for any help.0 -
My video sitemap is not being index by Google
Dear friends, I have a videos portal. I created a video sitemap.xml and submit in to GWT but after 20 days it has not been indexed. I have verified in bing webmaster as well. All videos are dynamically being fetched from server. My all static pages have been indexed but not videos. Please help me where am I doing the mistake. There are no separate pages for single videos. All the content is dynamically coming from server. Please help me. your answers will be more appreciated................. Thanks
Technical SEO | | docbeans0 -
Have I constructed my robots.txt file correctly for sitemap autodiscovery?
Hi, Here is my sitemap: User-agent: * Sitemap: http://www.bedsite.co.uk/sitemaps/sitemap.xml Directories Disallow: /sendfriend/
Technical SEO | | Bedsite
Disallow: /catalog/product_compare/
Disallow: /media/catalog/product/cache/
Disallow: /checkout/
Disallow: /categories/
Disallow: /blog/index.php/
Disallow: /catalogsearch/result/index/
Disallow: /links.html I'm using Magento and want to make sure I have constructed my robots.txt file correctly with the sitemap autodiscovery? thanks,0 -
Are Tags in Blogs good?
Hi Is adding tags into Blogs a good idea? Does it help with SEO at all? For example blog/?tag=/Invoicing-Software Will that help us get ranked for Invoicing Software? Regards Andrew
Technical SEO | | Studio330 -
Is there a suggested limit to the amount of links on a sitemap?
Currently, I have an error on my moz dashboard indicating there are too many links on one of my pages. That page is the sitemap. It was my understanding all internal pages should be linked to the sitemap. Can any mozzers help clarify the best practice here? Thanks, Clayton
Technical SEO | | JorgeUmana0 -
Sitemap.xml - autogenerated by CMS is full of crud
Hi all, hope you can help. the Magento ecommerce system I'm working with autogenerates sitemap.xml - it's well formed with priority and frequency parameters. However, it has generated lots of URLs that are pointing to broken pages returning fatal erros, duplicate URLs (not canonicals), 404s etc I'm thinking of hand creating sitemap.xml - the site has around 50 main pages including products and categories, and I can get the main page URLs listed by screaming frog or xenu. Then I'll have to get into the hand editing the crud pages with noindex, and useful duplicates with canonicals. Is this the way to go or is there another solution thanks in advance for any advice
Technical SEO | | k3nn3dy30 -
Good organic rankings but no shopping rankings
My site www.hair1direct.co.uk comes up well in the organic rankings but when I select shopping in Google it does appear. Why would this be? Thanks James
Technical SEO | | avecsys0