Can we listed URL on Website sitemap page which are blocked by Robots.txt
-
Hi,
I need your help here.
I have a website, and few pages are created for country specific. (www.example.com/uk).
I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page)
I really appreciate your help.
Thanks,
Nilay
-
if the content is of benefit to the user then include them in your navigation. Why are you blocking them in the first place, duplicate content?
-
Hi Zora,
But the pages which i have blocked are only visible in specific country. Addition, i have blocked theme also, so you think it's good to put those url on website or link them in website?
-
Hi Zora,
Thanks for your time.
-
Hi Jarno,
Thanks for your time.
Should i put URLs anywhere in my websites which are being excluded by Robot.txt?
Thanks,
Nilay
-
Hi Nilay,
I actually did this yesterday by accident.
I recommend you remove the blocked pages from your XML sitemap, otherwise Google will display a "warning" after you submit it.As far as your HTML sitemap, it does not really matter.
I think you are okay to keep the links there. -
Nilay,
if you have blocked them from your robots.txt but you do enclude them in your sitemap xml or html then they will be indexed unless you enclode a meta robots in it with a noindex tag. If that tag is not in the pages and you enclude that page in your sitemap Google will feel it as important content in list it in the SERPs.
Hope this helps
Regards
Jarno
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it better to try and boost an old page that ranks on page #5 or create a better new page
Hello Everyone, We have been looking into our placements recently and see that one of our blog posts shows on page #5 for a popular keyword phrase with a lot of search volume. Lets say the keyword is "couples fitness ideas" We show on page 5 for a post /couples-fitness-ideas-19-tips-and-expert-advice/ We want to try and get on the first page for that phrase and wanted to know if it is better if we did one of the following: 1. Create a new page with over 100 ideas with a few more thousands of words. with a new url (thinking /couples-fitness-ideas) 2. Create a new page with a new url (thinking /couples-fitness-ideas) with the same content as the currently ranking post. We would want to do this for more freedom with layout and design of the page rather than our current blog post template. Add more content, let's say 100 more ideas. Then forward the old URL to the new one with a 301 redirect. 3. Add more content to the existing post without changing the layout and change the URL. Look forward to your thoughts
On-Page Optimization | | MobileCause0 -
My website is being opened by multiple URLs?
Dear Friends, I have a website which is being opened like example.comand example/index to same page. Is there any problem SEO point of view. By the way I have placed the Rel= canonical tag in source page. its working fine but
On-Page Optimization | | docbeans0 -
How can I drive organic traffic to a specific landing page?
HI, I have a site which is attracting traffic for my target keywords but to the wrong pages. I usually create a series of articles on the topic (10-15) an start getting organic traffic, but I have not been able to drive the traffic to the main page for that topic. How can I get the main page rank over sub pages? Thanks in advance!
On-Page Optimization | | Rajaindiain0 -
Too many page links warning... but each link has canonical back to main page? Is my page OK?
The Moz crawl warns me many of my pages have too many links, like this page http://www.webjobz.com/jobs/industry/Accounting ...... has 269 links but many of the links are like this /jobs/jobtitles/Accounting?k=&w=3&hiddenLocationID=463170&depth=2 and are used to refine search criteria.... when you click on those links they all have a canonical link back to http://www.webjobz.com/jobs/industry/Accounting Is my page being punished for this? Do I have to put "no follow" tags on every link I do not want the bots to follow and if I do so is Roger (moz bot) not going to count this as a link?
On-Page Optimization | | Webjobz0 -
Optimize URL
Hello, My website have been running over five years. I have just reviewed and seen some URLs had not good. It is http://www.vietnamvisacorp.com/faqs/who-need-visa-to-vietnam---1.html, containing characters "---1". Should I remove unnecessary characters "---"?. Thanks for any advice!
On-Page Optimization | | JohnHuynh0 -
What kind of pages are they?
Hi all, after making an analysis of my website the tool has found two pages that I don't know what refers to: /?page id=2058 (only one page of this kind) /?attachment id= (only one page of this kind) does anyone know what kind of pages they could be? Do the have relevancy regading SEO? The plattform is Wordpress. Thanks!
On-Page Optimization | | juanmiguelcr0 -
Some of my pages are ranking for terms which I want other pages to rank for. What can I do to effectively switch the ranking?
Some of the pages are ranking for terms I have optimised other pages for. The pages which are ranking are quite rightly falling, because they aren't optimised for the terms they're showing for. However, I have pages which are optomised for those terms. How do I switch the SERPS to the page I want?
On-Page Optimization | | GlobalLingo1 -
On page Analysis... how does seomoz choose to match a keyword to a url?
On my "On Page" analysis report it show me a couple pages that have D ratings... one key phrase is "my destination lodging" and it is matching it up with a url like mydomain.com/my-destination-hotels.html. It says I'm ranking 37th for this term.... Is it just suggesting that I optimize this url because it is already ranking for that term? So should I create a different page with a different url to better match this phrase? Thanks for any ideas.
On-Page Optimization | | PillarMarketing0