Can we listed URL on Website sitemap page which are blocked by Robots.txt
-
Hi,
I need your help here.
I have a website, and few pages are created for country specific. (www.example.com/uk).
I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page)
I really appreciate your help.
Thanks,
Nilay
-
if the content is of benefit to the user then include them in your navigation. Why are you blocking them in the first place, duplicate content?
-
Hi Zora,
But the pages which i have blocked are only visible in specific country. Addition, i have blocked theme also, so you think it's good to put those url on website or link them in website?
-
Hi Zora,
Thanks for your time.
-
Hi Jarno,
Thanks for your time.
Should i put URLs anywhere in my websites which are being excluded by Robot.txt?
Thanks,
Nilay
-
Hi Nilay,
I actually did this yesterday by accident.
I recommend you remove the blocked pages from your XML sitemap, otherwise Google will display a "warning" after you submit it.As far as your HTML sitemap, it does not really matter.
I think you are okay to keep the links there. -
Nilay,
if you have blocked them from your robots.txt but you do enclude them in your sitemap xml or html then they will be indexed unless you enclode a meta robots in it with a noindex tag. If that tag is not in the pages and you enclude that page in your sitemap Google will feel it as important content in list it in the SERPs.
Hope this helps
Regards
Jarno
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ranking dropped after change single page url, should I change it back?
I was making updates to the content on the following page, and a few days later dropped from #2 SERP ranking to 50+. Things I checked: Yes, 301 redirect was implemented right away. After publishing, I manually requested indexing in search console. Right after publishing I re-submitted the sitemap manually and Google said they had not crawled it in 9 days. My question: should I change the URL back to the old one, or give it a little more time (especially since I re-submitted sitemap) Original URL: https://www.travelinsurancereview.net/plans/travel-medical/ New URL: https://www.travelinsurancereview.net/plans/travel-medical-insurance/
On-Page Optimization | | DamianTysdal0 -
Many of my pages are showing up as 4xx errors because they require a login/subscription to view. How can I fix this?
Should these pages be redirecting to a login page? Or does there need to be something other than an Access Denied message? I feel like these are bringing the overall site score down.
On-Page Optimization | | CaraMc0 -
Is it safe to add a Backlink on all pages on another website?
I got a few old websites and I would like to add a do-follow backlink on all pages linking to a new website. Is this SEO safe to do so?
On-Page Optimization | | hakhan2010 -
URL Structure Category Pages -Current Moz Friday-
Hello,
On-Page Optimization | | _Heiko_
regarding #15 of the last moz friday I have a question: http://moz.com/blog/15-seo-best-practices-for-structuring-urls What would you prefer if the lenght of the URL will be still under 60 characters and you have an example like this: Let's call it a specific page in a category. As I like the old shoe examples: You have a page about red shoes in your shoe category. Which URL would you prefer: a) www.mydomain.com/shoes/red-shoe b) www.mydomain.com/shoes/red Personally I would prefer a) or would you already consider this as spammy? My real example is not that trivial like the shoe example and the categories will be in plural and the specific pages always in singular (like in the example shoes vs shoe). c) would be to put it independently from the side structure on www.mydomain.com/red-shoe - but personally I have the experience that a) or b) will help the rankings of the category page if you have the specific pages in the same subfolder. What's your opinion on this?1 -
Page authority still on 1 after url change and 301 redirect
Hi Moz analytics suggestion to help ranking is to have a keyword or phrase in the url so I advised a client to do this they changed one of their pages urls, this page previously had a page authority of 26 since the change its gone down to 1.
On-Page Optimization | | genkee
I advised them that they must do a 301 from the old page but they took a few weeks to do this, would this of affected it why is it not showing up yet its been 3 weeks now, since the 301 and 5 weeks since the url change.0 -
Website went from page 10 to 31 for a keyword
Hey Everyone I hope someone could really help me here. My website: goo.gl/5fLqv, it is an ecommerce website and just in the last few days for one my keywords "Wedding Favours" it dropped to page 31 from page 10. Could someone help me please on how to resolve this? Thanks
On-Page Optimization | | VIVOWeb0 -
Too many page links warning... but each link has canonical back to main page? Is my page OK?
The Moz crawl warns me many of my pages have too many links, like this page http://www.webjobz.com/jobs/industry/Accounting ...... has 269 links but many of the links are like this /jobs/jobtitles/Accounting?k=&w=3&hiddenLocationID=463170&depth=2 and are used to refine search criteria.... when you click on those links they all have a canonical link back to http://www.webjobz.com/jobs/industry/Accounting Is my page being punished for this? Do I have to put "no follow" tags on every link I do not want the bots to follow and if I do so is Roger (moz bot) not going to count this as a link?
On-Page Optimization | | Webjobz0 -
What is on page links?
Hi - i would like to know exactly what an on page link is? i understand the linking system however cant work what exactly what an on page link is? Thanks
On-Page Optimization | | OasisLandDevelopment0