Can we listed URL on Website sitemap page which are blocked by Robots.txt
-
Hi,
I need your help here.
I have a website, and few pages are created for country specific. (www.example.com/uk).
I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page)
I really appreciate your help.
Thanks,
Nilay
-
if the content is of benefit to the user then include them in your navigation. Why are you blocking them in the first place, duplicate content?
-
Hi Zora,
But the pages which i have blocked are only visible in specific country. Addition, i have blocked theme also, so you think it's good to put those url on website or link them in website?
-
Hi Zora,
Thanks for your time.
-
Hi Jarno,
Thanks for your time.
Should i put URLs anywhere in my websites which are being excluded by Robot.txt?
Thanks,
Nilay
-
Hi Nilay,
I actually did this yesterday by accident.
I recommend you remove the blocked pages from your XML sitemap, otherwise Google will display a "warning" after you submit it.As far as your HTML sitemap, it does not really matter.
I think you are okay to keep the links there. -
Nilay,
if you have blocked them from your robots.txt but you do enclude them in your sitemap xml or html then they will be indexed unless you enclode a meta robots in it with a noindex tag. If that tag is not in the pages and you enclude that page in your sitemap Google will feel it as important content in list it in the SERPs.
Hope this helps
Regards
Jarno
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hi i have a few pages with duplicate content but we've added canonical urls to them, but i need help understanding what going on
hi google is seeing many of our pages and dupliates but they have canonical url on there https://www.hijabgem.com/index.php/maxi-shirt-dress.html has tags https://www.hijabgem.com/maxi-shirt-dress.html
On-Page Optimization | | hijabgem
has tagshttps://www.hijabgem.com/index.php/quickview/index/view/id/4693
has tags
my question is which page takes authority?and are they setup correct, can you have more than one link rel="canonical" on one page?0 -
To avoid the duplicate content issue I have created new urls for that specific site I am posting to and redirecting that url to the original on my site. Is this the right way to do it?
I am trying to avoid the duplicate content issue by creating new urls and redirecting them to the original url. Is this the proper way of going about it?
On-Page Optimization | | yagobi210 -
Robots.txt Question for E-Commerce Sites
Hi All, I have a couple of e-commerce clients and have a question about URLs. When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout I'm not sure that I want these indexed. Could I be causing any harm/danger if I add this to the robots.txt file? /*? Any suggestions welcome! Gavin
On-Page Optimization | | IcanAgency0 -
Why is my contact us page ranking higher than my home page?
Hello, It doesn't matter what keyword I put into Google (when I'm not signed in and have cleaned down my browsing history) the contact us page ranks higher than the home page. I'm not sure why this is, the home page has a higher page authority, more links and more social media shares, the website is an established one. When I have checked Google Analytics my home page gets more people landing on it than the contact us page. It looks like people are ignoring the contact us page and scrolling down until they find the home page. I'd appreciate any help or advice you might have. Thank you.
On-Page Optimization | | mblsolutions2 -
Which redirect to use when redirecting to https page from http page
I have one form under https which is redirected from the regular http page. this site was not made by me and I am trying to understand if the way it was redirected using 302 redirect is a problem Thanks
On-Page Optimization | | ciznerguy0 -
Image URL's have knocked my sub-pages down (WP)
I had most of my keywords within the top 10 for this site, some were even ranking in the top 5. For a possible minor boost, more-so to cover all the bases, I decided to add images to all of the pages, and they were uploaded as a gallery with most of the image file names being the same as the keyword. Thus, url's were created with our targeted phrases, extending off of the corresponding sub-page. After that, Google quickly picked up the url's to the images and began indexing them, when that occurred the sub-page which was to be the landing page, quickly tanked. Nothing else on-site changed besides the uploading of the images, so I'm sure they're conflicting and for whatever reason Google can't decide which page to index. The page that contains the images used, or the actual intended landing page. With WP I didn't see a way to not have them link to anything at all, and just be static when using a gallery, stock at least. So, my question is how can I quickly alleviate this problem and what should I do in the future to avoid this? I believe if I change link thumbnails to image file instead of attachment page, that should fix the issue... Then, I'll have dead URL's which I suppose I should 301 to the sub-page. Alternatively, is there a better solution that will work, I was also thinking about no-indexing the attachment URL's, but that doesn't seem to be an option.
On-Page Optimization | | JayAdams320 -
Website tables
Hi guys , my website www.starpluservices.com on google first page for the past 2 years and we were page rank 4, 2 months ago we changed all page titles and Keewords after that the page rank droped to 1 and we are not anymore in page 1 in google, we have done all this changes to target another keyword Office Cleaning London, now after 2 months I had 3 quotes for SEO , and 2 SEO companies told me that I need a new website because my one was done with tables and the other company told me that if I still on first page with some keywords andon the 2 page with cleaning companies to dont make a new website just update my one!! Could anyone let me know what should I do? Regards Sergio
On-Page Optimization | | starplus0 -
How can I get on-page analysis reports ?
I started trying out seomoz' facilities as a way to give myself a checklist of things to do-and-don't while learning SEO. Faced with a ton of new and interesting information, I started optimizing my home page and writing some content that can tell a story to people in my business. So I started using on-page analysis tools. But the campaign opening screen is telling me to wait for reports approx. two weeks. There are some reports ( not so good ones, a lot to improve ) but not the one I need most: the on-page analysis one. I set up two campaigns and I have the same problem for both. Am I doing something wrong ? Can I change my pages to make this work ?
On-Page Optimization | | wiebe0