Can we listed URL on Website sitemap page which are blocked by Robots.txt
-
Hi,
I need your help here.
I have a website, and few pages are created for country specific. (www.example.com/uk).
I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page)
I really appreciate your help.
Thanks,
Nilay
-
if the content is of benefit to the user then include them in your navigation. Why are you blocking them in the first place, duplicate content?
-
Hi Zora,
But the pages which i have blocked are only visible in specific country. Addition, i have blocked theme also, so you think it's good to put those url on website or link them in website?
-
Hi Zora,
Thanks for your time.
-
Hi Jarno,
Thanks for your time.
Should i put URLs anywhere in my websites which are being excluded by Robot.txt?
Thanks,
Nilay
-
Hi Nilay,
I actually did this yesterday by accident.
I recommend you remove the blocked pages from your XML sitemap, otherwise Google will display a "warning" after you submit it.As far as your HTML sitemap, it does not really matter.
I think you are okay to keep the links there. -
Nilay,
if you have blocked them from your robots.txt but you do enclude them in your sitemap xml or html then they will be indexed unless you enclode a meta robots in it with a noindex tag. If that tag is not in the pages and you enclude that page in your sitemap Google will feel it as important content in list it in the SERPs.
Hope this helps
Regards
Jarno
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Too many links per page? Double navigation on every page...
I have a client with navigation across the top of each page plus the same nav links in a sidebar on every page. Can that duplication (or the sheer number of links) on each page have a negative ranking factor?
On-Page Optimization | | brm20170 -
Page/Website Structure
Hello again Mozzers, We have a category, lets call it widgets. Within widgets are about a hundred or so products. For usability my predecessor made the following layout Widgets Main Cateogry - Links off homepage - (no content just links to the 3 sub-categories)
On-Page Optimization | | ATP
- Widgets by Resolution
---- About 20 subcategories
eg. 0.1 Resolution widgets
0.2 resolution widgets
- Widgets by Capacity
---- About 20 subcategories
eg. 1 capacity widgets
2 capacity widgets
- Widgets by Type
---- About 12 subcategories This was a major improvement from a userbility perspective as it made a very complex product range navigatable by the major features or basic type. However, as you can imaging we now have 60+ very similiar pages all displaying very similiar products a nightmare for SEO. It also isnt ideal for user navigation as it take too many clicks to get to the products. I propose the following fix, and i wanted your opinion. Widget Main Category - Link from homepage (Consolidated with Widgets by Type)
-300 Words of content
-Links to the 12 Sub-type Catoregies (These are pages i can fill with content + products. This would give me a more ordinary structure of which I can focus each page to a keyword) The tricky part comes with incorporating the capacity and resolution options. 1 Browse Capacity Page
(20 sub categories all the same except capacity quantity & products)
1 Browse by Resolution Page
(20 sub categories all the same except resolution value & products) The owner want them, I was going to link from the main widgets page to each of these to give the customer the option. What I can't decide is how to deal with them from an SEO point of view. Should they be no-followed? canonicaled? Can there be any advantage to having so many pages covering slightly different variations or as i suspect it is dangerous to the overall health of the site. To complicate things further, Canonical tags may not be an option due to an old magento version running that doesnt support them. Is there an alternative way around? As always many thanks.0 -
Short URL's vs Optimised URL's
Howdy Mozzers! What are your thoughts on short URL's vs Optimised URL's. For example if a website currently sells wood furniture and wants to target the keyword "Wood Furniture For Sale", which URL would be preferable: Short URL: www.domain.com/wood-furniture Optimised URL: www.domain.com/wood-furniture-for-sale The website also uses facet navigation and selected attributes are added in a fixed order sequence after the category. For example if Cane is selected as wood type: Short URL: www.domain.com/wood-furniture/Cane Optimised URL: www.domain.com/wood-furniture-for-sale/Cane Which one do you prefer (between the short URL and optimised URL) and why? Cheers! MozAddict
On-Page Optimization | | MozAddict0 -
If I want to rank well on one keyword would it be better to optimize multiple pages on the website for the keyword or should I only optimize one page for that keyword?
If I want to rank well on one keyword would it be better to optimize multiple pages on the website for the keyword or should I only optimize one page for that keyword?
On-Page Optimization | | CustomOnlineMarketing0 -
How many keywords max can I optimize each page for?
I don't want to over optimize by doing 1 keyword per 1 page, but then if I do more, seomoz on-page tool report doesn't give an A grade for each keyword I optimize. I usually optimize for max 3 keywords that are very closely related, meaning they use the same words. Ex. dentist los angeles, los angeles dentist, dentist in los angeles Am I on the right track or what's your recommendation? Should I create different landing pages for each keyword?
On-Page Optimization | | sub90900 -
URL for location pages
Hello all We would like to create clean, easy URLs for our large list of Location pages. If there are a few URLs for each of the pages, am I right when I'm saying we would like this to be the canonical? Right now we would like the URL to be: For example
On-Page Optimization | | Ferguson
Domain.com/locations/Columbus I have found some instances where there might be 2,3 or more locations in the same city,zip. My conclusion for these would be: adding their Branch id's on to the URL
Domain.com/locations/Columbus/0304 Is this an okay approach? We are unsure if the URL should have city,State,zip for SEO purposes?
The pages will have all of this info in it's content
BUT what would be best for SEO and ranking for a given location? Thank you for any info!0 -
SEO for One Page Websites
Hi Are there any SEO guidelines for "one page websites". I'm looking into the 'benefit' it might have in combination with exact match URLs. Many thanks in advance.
On-Page Optimization | | Partouter2 -
Can someone please help me identify where all these URLS to my homepage are coming from?
Hi. I installed the SEOmoz toolbar for Firefox, and analyzed my home page, then clicked on 'get a full site analysis at Site Explorer'. This is what came up: http://www.opensiteexplorer.org/www.frs-solutions.com%252Fcontent%252Fhome/a!links?src=mb I hope that link works. If not, the URL is www.frs-solutions.com Anyway, there are about 57 different URLS within my site all pointing to my homepage! I have no idea where they are coming from. Can someone with an experienced eye take a quick look and tell me what I might be up against? Thank you!
On-Page Optimization | | aprilm-1890400