Can we listed URL on Website sitemap page which are blocked by Robots.txt
-
Hi,
I need your help here.
I have a website, and few pages are created for country specific. (www.example.com/uk).
I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page)
I really appreciate your help.
Thanks,
Nilay
-
if the content is of benefit to the user then include them in your navigation. Why are you blocking them in the first place, duplicate content?
-
Hi Zora,
But the pages which i have blocked are only visible in specific country. Addition, i have blocked theme also, so you think it's good to put those url on website or link them in website?
-
Hi Zora,
Thanks for your time.
-
Hi Jarno,
Thanks for your time.
Should i put URLs anywhere in my websites which are being excluded by Robot.txt?
Thanks,
Nilay
-
Hi Nilay,
I actually did this yesterday by accident.
I recommend you remove the blocked pages from your XML sitemap, otherwise Google will display a "warning" after you submit it.As far as your HTML sitemap, it does not really matter.
I think you are okay to keep the links there. -
Nilay,
if you have blocked them from your robots.txt but you do enclude them in your sitemap xml or html then they will be indexed unless you enclode a meta robots in it with a noindex tag. If that tag is not in the pages and you enclude that page in your sitemap Google will feel it as important content in list it in the SERPs.
Hope this helps
Regards
Jarno
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it better to keep a glossary or terms on one page or break it up into multiple pages?
We have a very large glossary of over 1000 industry terms on our site with links to reference material, embedded video, etc. Is it better for SEO purposes to keep this on one page or should we break it up into multiple pages, a different page for each letter for example? Thanks.
On-Page Optimization | | KenW0 -
How to fix this URL?
Moz crawl test only show one URL indexed for the website - http://ravisharrma.com/ Site:http://ravisharrma.com/ shows 3 URLs and how to fix URLs for the website?
On-Page Optimization | | Dan_Brown10 -
Sitemap error is reported when using a sitemap-index generated by Yoast
I've installed the Yoast SEO Plugin for wordpress and I've setup the sitemaps using it. I saw the tool has generated the Sitemap index file http://www.phraseexpander.com/sitemap_index.xml with different indexes for posts and pages I've submitted that to google and it's indexed. When I use Seoquake to check my website, I see that it says that the sitemap is missing (in fact http://www.phraseexpander.com/sitemap.xml) is returning 404. Shall I fix that? Shall I do a 301 redirect in my .htaccess file to http://www.phraseexpander.com/sitemap_index.xml Thanks.
On-Page Optimization | | nagar0 -
Website pages missing from seomoz crawl
Hi! I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing? Thanks!
On-Page Optimization | | ovieira0 -
Rewrite URL for big website
My website is currently have over 1000 000 links indexed by Google , i'm going to rewrite all of these links, is this crazy ? Can someone give me a checklist to avoid going wrong way ? My website: http://www.webtretho.com/forum/ My current url format: webtretho.com/forum/f[forum_id]/[thread-title-thread_id/ Ex: Old URL: http://www.webtretho.com/forum/f90/chia-se-dau-hieu-mang-thai-1357342-new/ New URL: http://www.webtretho.com/forum/chuan-bi-mang-thai/chia-se-dau-hieu-mang-thai-1357342-new/
On-Page Optimization | | firstjames0 -
Break-up content into individual pages or keep on one page
I am working on a dental website. Under menu item "services" lists everything he does like.. Athletic Sports Guards
On-Page Optimization | | Czubmeister
An athletic sports guard is a resilient plastic appliance that is worn to protect the teeth and gum tissues by absorbing the forces generated by traumatic blows during sports or other activities. Digital X-Rays We use state of the art digital x-rays and digital cameras to help with an accurate diagnosis of any concerns. Digital Imaging On initial visits, and recall visits, we take a series of digital photographs to aid us in diagnosis as well as to give you a close-up view of your mouth and any oral conditions. Smile Makeovers
We offer a number of different options including bleaching, bonding, porcelain veeners, and in some cases, implants and/or orthodontic care is utilized in our smile makeover planning. Nitrous oxide for your Comfort Would it be better to break these services up into individual pages? I was thinking I would because then I could add more pictures and expand on the topic and try to get an "A" grade on each page. I'm not sure how I could rank a page if I have 35 services listed on the page. That would be an awfully big H1! Suggestions?0 -
How can I get on-page analysis reports ?
I started trying out seomoz' facilities as a way to give myself a checklist of things to do-and-don't while learning SEO. Faced with a ton of new and interesting information, I started optimizing my home page and writing some content that can tell a story to people in my business. So I started using on-page analysis tools. But the campaign opening screen is telling me to wait for reports approx. two weeks. There are some reports ( not so good ones, a lot to improve ) but not the one I need most: the on-page analysis one. I set up two campaigns and I have the same problem for both. Am I doing something wrong ? Can I change my pages to make this work ?
On-Page Optimization | | wiebe0 -
Where to Put Content For Product Pages - How To Structure Website?
Currently we have 300+ products. We do not have a CMS or Ecommerce site at the time being for certain reasons. Currently our site is set up with content on almost every single page. The main catagory page, explains everything on the main page, then our products page has a lot of text too. But right now, it seems as if our main pages are only ranking. In the near future I will be using a cms and purchasing a template. I noticed most Ecommerce style websites have just the product with the name and price, then when they click on the product it brings you to that page with a brief product description and some photos. My question is, does each page need content? Or can just the product page itself have content? For example, say we have a link to SHOES. Then the shoes page displays dress, casual and athletic. Then the athletic page brings you to a page with, running, tennis, cross training shoes, and so forth. Is it best to write content on this main catagory page? If so, how much? Or should we focus on putting content on the actual page of the individual product? Along with pictures and specifications? I know Content is Key and we are doing pretty well at that, however, I am starting to wondering if we have to much content or too similar content. What is the best structure to try and recieve GREAT organic rankings?
On-Page Optimization | | hfranz0