Can we listed URL on Website sitemap page which are blocked by Robots.txt
-
Hi,
I need your help here.
I have a website, and few pages are created for country specific. (www.example.com/uk).
I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page)
I really appreciate your help.
Thanks,
Nilay
-
if the content is of benefit to the user then include them in your navigation. Why are you blocking them in the first place, duplicate content?
-
Hi Zora,
But the pages which i have blocked are only visible in specific country. Addition, i have blocked theme also, so you think it's good to put those url on website or link them in website?
-
Hi Zora,
Thanks for your time.
-
Hi Jarno,
Thanks for your time.
Should i put URLs anywhere in my websites which are being excluded by Robot.txt?
Thanks,
Nilay
-
Hi Nilay,
I actually did this yesterday by accident.
I recommend you remove the blocked pages from your XML sitemap, otherwise Google will display a "warning" after you submit it.As far as your HTML sitemap, it does not really matter.
I think you are okay to keep the links there. -
Nilay,
if you have blocked them from your robots.txt but you do enclude them in your sitemap xml or html then they will be indexed unless you enclode a meta robots in it with a noindex tag. If that tag is not in the pages and you enclude that page in your sitemap Google will feel it as important content in list it in the SERPs.
Hope this helps
Regards
Jarno
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I utilize URL re-writes to include keywords and other optimised page elements on my website?
Hi there, I am working on a medical recruitment website: https://wave.com.au/ I have noticed that our Job Search and Job Listings section of the website is lacking a little bit in terms of SEO optimisation. For example, at the moment this is our search page (with locum work type selected): https://wave.com.au/search-locum/results If you add a location, for example, NSW and then click search again, it updates the URL to: https://wave.com.au/search-locum/results?LocumSearchForm_Location[]=NSW&action_doSearch=Search+jobs I did a check what some competitors and leaders in the recruitment industry were doing and I came across: https://www.ochrerecruitment.com.au/jobs/anaesthetics/new-south-wales/locum/ If you click the different filters/options, it updates the URL to include more clearly defined categories. Some websites would even have a H1 heading tag that would update based on the filters/options you selected. Should I set up a set of URL re-writes and re-structure my website a little bit so that dynamic URLs change to static etc.? Does anyone have any best practice knowledge in regards to this? I have been referencing the following article: https://moz.com/blog/dynamic-urls-vs-static-urls-the-best-practice-for-seo-is-still-clear
On-Page Optimization | | Wavelength_International0 -
Changing the url of a page
Hello. I would like to change the url of a page. It currently has very few inbound links. I would set up a 301 redirect to the new url. Is there anything else I should take into account before changing the url? Is there a downside to changing a url? Do inbound links carry the same value when a 301 redirect is involved? Thank you!
On-Page Optimization | | nyc-seo0 -
Home page keyword effecting internal page ranking
Hello, My client has a second keyword for the home page that is competitive. The home page is not being ranked for this keyword. Instead, an internal category page is ranking. This internal category page is more relevant than the home page - it shows the categories for the actual products that this term refers to. But everyone around us in Google's page results has far more backlinks than the internal page, and we're all heavily optimized for this term. My question is, is it safe to pull the second term off of the home page or is this internal page strong because it is somehow being strengthened by the home page optimization?
On-Page Optimization | | BobGW0 -
Lots of Listings on Page 2 SERP, but none on the first?
For one of our most important keywords we have several pages ranking highly on the second page of search results in google. Yet none on the first page, is there anyway to concentrate our rankings to bump one to the first page? We would rather have even just one link on the first page than the 4 or 5 we have on the second now.
On-Page Optimization | | absoauto0 -
Jquery in top of page vs text on bottom page
Is it the best way to use jquery sliders on the top of your page to still get all your text above the fold and score in search engines? for example: http://www.wolf-howl.com/wp-conte... is much better to score high ranks in search engines than http://www.wolf-howl.com/wp-conte... ?? Thanks!
On-Page Optimization | | HMK-NL0 -
Dupelicate content home page and custom page question
I am working on a website that got hit by the penguin update. Didn't get hit terribly bad, but dropped from number one to number 9. As I'm going through the pages, the theme and content is a mess. To give an example, say the site is about custom colored marbles. The main page content covers custom colored marbles, custom promotional marbles, custom glass marbles, etc. Custom colored marbles is mentioned and covered on all pages, which I am going back and trying to make each page theme specific. There is also a custom page, so I am at a cross roads on how best to employ the focus of the custom page and the home page. I am thinking the home page should emphasize colored marbles, and the custom page should emphasize custom colored marbles. My fear is that making such a drastic change will bounce the site completely off front page and that it will take time for the custom page to come up in rankings. AS it stands now I am confused as to how it even ranks on first page as there's two pages with custom colored marbles emphasis. Id like to clean this up as much as possible so there are no big hits with future google updates, but I don't want the site to drop off either as that would be hard to explain to the owner. Yeah, we are cleaning up your site and making it google compliant and in so doing you no longer rank on first page. That won't put food on the table. Thanks for any advise on this.
On-Page Optimization | | anthonytjm0 -
My website is saying I have duplicate page content and page title. How do I fix it?
Hi, I created a website on webstarts.com. After I launched it then ran a scan through SEO it says I have duplicate page content and page title. The 2 pages it is reading are technically the same page. www.mobilemowermedicsinc.com and www.mobilemowermedicsinc.com/index . I am unsure how to get rid of on of these as it keeps saying this is an error in the SEO scan. Could someone please advise me of what to do from here. Thanks!
On-Page Optimization | | bcarp880 -
Wordpress categories tags and robots.txt
I am relatively new at this and see a variety of people that seem to disagree on if you should block google from indexing category and tag pages through robot.txt or no-follow because of google viewing it as duplicate content. I tryst this communities answers over the web at large obviosly, so what do you all think? Thanks, Steven
On-Page Optimization | | sfmatthews0