Can we listed URL on Website sitemap page which are blocked by Robots.txt
-
Hi,
I need your help here.
I have a website, and few pages are created for country specific. (www.example.com/uk).
I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page)
I really appreciate your help.
Thanks,
Nilay
-
if the content is of benefit to the user then include them in your navigation. Why are you blocking them in the first place, duplicate content?
-
Hi Zora,
But the pages which i have blocked are only visible in specific country. Addition, i have blocked theme also, so you think it's good to put those url on website or link them in website?
-
Hi Zora,
Thanks for your time.
-
Hi Jarno,
Thanks for your time.
Should i put URLs anywhere in my websites which are being excluded by Robot.txt?
Thanks,
Nilay
-
Hi Nilay,
I actually did this yesterday by accident.
I recommend you remove the blocked pages from your XML sitemap, otherwise Google will display a "warning" after you submit it.As far as your HTML sitemap, it does not really matter.
I think you are okay to keep the links there. -
Nilay,
if you have blocked them from your robots.txt but you do enclude them in your sitemap xml or html then they will be indexed unless you enclode a meta robots in it with a noindex tag. If that tag is not in the pages and you enclude that page in your sitemap Google will feel it as important content in list it in the SERPs.
Hope this helps
Regards
Jarno
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What to do about pages I have deleted?
I have been working through the dead links on my page and recreating the page with new content for those pages that it still makes sense to have on the site. But I have a few that were just changes of the title, spelling mistakes or other ways of saying the same thing In other words I created a page called "areas of the UK we cover" but decided to change it to "areas covered" However, I must have created links to this page and now it is a dead link with a page authority of 19 I think it would be spammy to have two pages, one called "areas covered" and the other called "areas of the UK we cover. It's not a disallow in Robots.txt because the page does not exist Please note I do not have access to the header to add code for a 301 redirect. I'm still using webs.com but not for new sites. I also have a page called singing telegrams london, that I changed from singagrams london. These are two words for the same thing but they are two very different keywords would it be ok to recreate this page and create content for singagrams london. Help is much appreciated
On-Page Optimization | | singingtelegramsuk0 -
How can I drive organic traffic to a specific landing page?
HI, I have a site which is attracting traffic for my target keywords but to the wrong pages. I usually create a series of articles on the topic (10-15) an start getting organic traffic, but I have not been able to drive the traffic to the main page for that topic. How can I get the main page rank over sub pages? Thanks in advance!
On-Page Optimization | | Rajaindiain0 -
Does having landing page text beneath the products at the base of the page hinder SEO?
I have a site that is capable of hosting the landing page description either above the products under the H1 or below them at the bottom of the page before the footer. I have always chosen to keep the text "above the fold" as presumably this would be crawled sooner in relation to the rest of the page content than had it been at the bottom. However, this means that I can only really write just a few sentences for each landing page - otherwise the products would shift further down the page - and I don't think this is good from a UX POV. Question: If I move the bulk of my landing page descriptions to the text snippet located underneath the products, could this negatively affect my SEO? Text at the bottom of the page is obviously not significant for users, so is there a chance this could be seen as spam?
On-Page Optimization | | Silkstream0 -
Short URL's vs Optimised URL's
Howdy Mozzers! What are your thoughts on short URL's vs Optimised URL's. For example if a website currently sells wood furniture and wants to target the keyword "Wood Furniture For Sale", which URL would be preferable: Short URL: www.domain.com/wood-furniture Optimised URL: www.domain.com/wood-furniture-for-sale The website also uses facet navigation and selected attributes are added in a fixed order sequence after the category. For example if Cane is selected as wood type: Short URL: www.domain.com/wood-furniture/Cane Optimised URL: www.domain.com/wood-furniture-for-sale/Cane Which one do you prefer (between the short URL and optimised URL) and why? Cheers! MozAddict
On-Page Optimization | | MozAddict0 -
Missing meta descriptions on indexed pages, portfolio, tags, author and archive pages. I am using SEO all in one, any advice?
I am having a few problems that I can't seem to work out.....I am fairly new to this and can't seem to work out the following: Any help would be greatly appreciated 🙂 1. I am missing alot of meta description tags. I have installed "All in One SEO" but there seems to be no options to add meta descriptions in portfolio posts. I have also written meta descriptions for 'tags' and whilst I can see them in WP they don't seem to be activated. 2. The blog has pages indexed by WP- called Part 2 (/page/2), Part 3 (/page/3) etc. How do I solve this issue of meta descriptions and indexed pages? 3. There is also a page for myself, the author, that has multiple indexes for all the blog posts I have written, and I can't edit these archives to add meta descriptions. This also applies to the month archives for the blog. 4. Also, SEOmoz tells me that I have too many links on my blog page (also indexed) and their consequent tags. This also applies to the author pages (myself ). How do I fix this? Thanks for your help 🙂 Regards Nadia
On-Page Optimization | | PHDAustralia680 -
Source page leading to a 404 pages in reports
Hi everybody, I wonder how to find and quickly correct 404 errors in my crawl reports : SeoMoz says me "http://domain.com/this-page-is-dead" is 404, but I can't figure out a source page where a link to that url appears. I tried a google link:http://domain.com/this-page-is-dead request, with no more luck. I imagine the trick is trivial, but I need it 🙂 Moreover, why do not show a list of pages referring to this 404 page on reports ? Thanks, Loïc
On-Page Optimization | | mandinga0 -
Can I use the first sentence of my page content as a meta description tag as well?
I just want to copy my content on the page and use the first or as well the second sentence of the content self for my meta description tag. Is that OK? Or should the Meta description tag be different?
On-Page Optimization | | paulinap19830