Can we listed URL on Website sitemap page which are blocked by Robots.txt
-
Hi,
I need your help here.
I have a website, and few pages are created for country specific. (www.example.com/uk).
I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page)
I really appreciate your help.
Thanks,
Nilay
-
if the content is of benefit to the user then include them in your navigation. Why are you blocking them in the first place, duplicate content?
-
Hi Zora,
But the pages which i have blocked are only visible in specific country. Addition, i have blocked theme also, so you think it's good to put those url on website or link them in website?
-
Hi Zora,
Thanks for your time.
-
Hi Jarno,
Thanks for your time.
Should i put URLs anywhere in my websites which are being excluded by Robot.txt?
Thanks,
Nilay
-
Hi Nilay,
I actually did this yesterday by accident.
I recommend you remove the blocked pages from your XML sitemap, otherwise Google will display a "warning" after you submit it.As far as your HTML sitemap, it does not really matter.
I think you are okay to keep the links there. -
Nilay,
if you have blocked them from your robots.txt but you do enclude them in your sitemap xml or html then they will be indexed unless you enclode a meta robots in it with a noindex tag. If that tag is not in the pages and you enclude that page in your sitemap Google will feel it as important content in list it in the SERPs.
Hope this helps
Regards
Jarno
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robot.txt file issue on wordpress site.
I m facing the issue with robot.txt file on my blog. Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue. The search result shows that "A description for this result is not available because of this site's robots.txt – learn more." Any suggestion to over come with this issue
On-Page Optimization | | Mustansar0 -
Can horrific grammar and spelling in comments hurt the value of an otherwise great page?
I've got a website whose pages get lots of comments. Tons of activity, which I would think Google would like (and seems to like). However -- I just can't put this nicely -- most commenters are not very bright. Their grammar and spelling is horrific. These are not foreigners who lack English skills, they are just about all primarily English speakers and the site is 99% US traffic. It's a low-income segment of the population. So, I've been wondering recently if Google will mark down the value of the page due to the bad grammar and spelling in the comments, even if the page's content is otherwise very good and lengthy. I have read that they grammar and spelling into consideration when looking at the page, but would that include comments, or would they know they are comments and not judge a page on that? It would be a pain, but maybe I should I run all the comments at least through a spell checker? And manually fix their grammar? Problem is I get about 40 comments a day. And when I say bad grammar and spelling, I mean REALLY bad. Embarrassing.
On-Page Optimization | | bizzer0 -
No more map listings
Since July'12, i have seen our Google Maps listings drop. They just don't seem to be showing in searches anymore. Can anyone shed any light as to why ths may be?
On-Page Optimization | | RobSchofield0 -
How do i block an entire category/directory with robots.txt?
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now. The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc. I'm not really sure how i'd type it into the robots.txt file, or where to place the file. any help would be appreciated thanks
On-Page Optimization | | bricerhodes0 -
Keyword in URL?
I have a website that has been live for about 8yrs. I do not have any significant rankings for my main keywords but am now starting SEO on my site. I am contemplating changing the url to contain the main keyword prefixed by my brand name. Any views on the ranking benefits and or CTR benefits.
On-Page Optimization | | Johnnyh
Example:
Main Volume keyword - 'car leasing'
current url - www.bobleasing.co.uk (made up name) thinking of changing to - www.bobcarleasing.co.uk (made up name) Any advice would be much appreciated. John0 -
View all Page for Product Overview Pages
Hi everybody! We have an ecommerce site with product overview pages, where sometimes there are hundreds of products listed. Usually, we just display 30 and have a button where users can click to see 30 more - or all products listed at once. This is the overview page (as indexed in google): http://www.geschenkidee.ch/aussergewoehnliches.html
On-Page Optimization | | zeepartner
And this is the view-all page: http://www.geschenkidee.ch/aussergewoehnliches.html#all What should I do here? The product overview page will hardly generate more traffic by listing all products (because the overview page will rank for generic keywords, while the product keyword searches will be referred to the specific product pages themselves). I was originally thinking of using rel=canonical pointing to the view-all page. But this would just lead to longer load time. Should we just leave those overview pages or is there a best practice for how to deal with such pages? Thanks for your thoughts on this!0 -
Anchor text, same page, different kewords to same URLs
Could someone please tell how Google treats the use of anchor text from a single page when using different keywords that all point to the same URL. So for instance I am doing a blog post and use the following anchor text which all point to the same URL: Cool Widget >> www.domain.com/widget Awesome Widget >> www.domain.com/widget Mighty Widget >> www.domain.com/widget I have read that Google will only take noticeof the first one? Thanks
On-Page Optimization | | blagger0