Can we listed URL on Website sitemap page which are blocked by Robots.txt
-
Hi,
I need your help here.
I have a website, and few pages are created for country specific. (www.example.com/uk).
I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page)
I really appreciate your help.
Thanks,
Nilay
-
if the content is of benefit to the user then include them in your navigation. Why are you blocking them in the first place, duplicate content?
-
Hi Zora,
But the pages which i have blocked are only visible in specific country. Addition, i have blocked theme also, so you think it's good to put those url on website or link them in website?
-
Hi Zora,
Thanks for your time.
-
Hi Jarno,
Thanks for your time.
Should i put URLs anywhere in my websites which are being excluded by Robot.txt?
Thanks,
Nilay
-
Hi Nilay,
I actually did this yesterday by accident.
I recommend you remove the blocked pages from your XML sitemap, otherwise Google will display a "warning" after you submit it.As far as your HTML sitemap, it does not really matter.
I think you are okay to keep the links there. -
Nilay,
if you have blocked them from your robots.txt but you do enclude them in your sitemap xml or html then they will be indexed unless you enclode a meta robots in it with a noindex tag. If that tag is not in the pages and you enclude that page in your sitemap Google will feel it as important content in list it in the SERPs.
Hope this helps
Regards
Jarno
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I remove a high traffic page on my website?
For the last few years, a particular blog post on my site has gotten 3 times as much traffic than any other page, even the home page; however, the topic of the post is only moderately related to the website topic and I'm wondering if all that unrelated traffic is negatively effecting SEO for our primary keywords. Here's an example.... Site topic: Yoga retreats in Costa Rica (we want to attract people who are interested in booking a yoga retreat) Blog Topic: How to extend your visa in Costa Rica (it's related only because it's about Costa Rica and travel, and may help our visitors stay longer) Other Notes: In 4 years, visitors to that blog post have never converted. Blog post bounce rate is 56%, significantly higher than almost any other page Lots of comments on the blog post so visitors to it are engaged and find it very useful To get an accurate reading of interested visitors to the site, i always have to filter entrance visits to this post in my analytics because these users are not an accurate representation of the visitors we're trying to draw. My question: Because I get so much traffic from the blog post, which is about the visa renewal process, will Google consider the website less about yoga and more about visas? If so, will it make it more difficult to rank well for yoga in Costa Rica? Does Google say to itself, "Hey, this website can't be an authority about both yoga and visas in Costa Rica so we're going to consider it a visa site because of all the visits and engagement it gets for that topic." So should I remove the post or just leave it alone? It offers a lot of people valuable information so I would never delete it entirely, but would redirect it somewhere else. Thanks!
On-Page Optimization | | Cabaretti0 -
Correct robots.txt for WordPress
Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong? I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right. Thank you so much. FzSQkqB.jpg
On-Page Optimization | | AslanBarselinov1 -
When making content pages to a specific page; should you index it straight away in GSC or let Google crawl it naturally?
When making content pages to a specific page; should you index it straight away in GSC or let Google crawl it naturally?
On-Page Optimization | | Jacksons_Fencing0 -
Building a new page: What on-page SEO would you build in?
Hi all, Building a new page for a fairly competitive keyword. Need to make sure the on-page SEO is pretty top notch, because link building (including internal links) will be difficult. I've optimised the meta description, the alt tags and image names, and included the keyword in the Title Tags. Not a great deal I can do with regards to optimising for mobile or considering migrating to the AMP project because this is handled externally. What else would you suggest? Cheers in advance, Rhys
On-Page Optimization | | SwanseaMedicine1 -
Keyword Appearing on Home Page - Moz Page Grader
Hi Today I entered www.partydomain.co.uk through the Moz Page Grader and found that the Home Page is Ranked B. I noticed that an Area we could improve on is the amount of times we are using our main keyword "Fancy Dress" on the home page. Please can you take a look at www.partydomain.co.uk and scroll to the bottom of the page were the tabs are containing losts of content. I am thinking about removing all of thoose Tabs. Our Competitors dont have any content as such on the home page and are ranking higher than Party Domain for "fancy dress" What do you think ? remove all the tabs to be like the others that rank better? Or cut the text right down ? Thanks Adam
On-Page Optimization | | AMG1000 -
Why is the seomoz showing it crawled 3 pages when i only have 2 pages?
I had seomoz crawl my site. I only have 2 pages. The site url is www.autoinsurancefremontca.com.
On-Page Optimization | | Greenpeak0 -
New CMS system - 100,000 old urls - use robots.txt to block?
Hello. My website has recently switched to a new CMS system. Over the last 10 years or so, we've used 3 different CMS systems on our current domain. As expected, this has resulted in lots of urls. Up until this most recent iteration, we were unable to 301 redirect or use any page-level indexation techniques like rel 'canonical' Using SEOmoz's tools and GWMT, I've been able to locate and redirect all pertinent, page-rank bearing, "older" urls to their new counterparts..however, according to Google Webmaster tools 'Not Found' report, there are literally over 100,000 additional urls out there it's trying to find. My question is, is there an advantage to using robots.txt to stop search engines from looking for some of these older directories? Currently, we allow everything - only using page level robots tags to disallow where necessary. Thanks!
On-Page Optimization | | Blenny0 -
What are the benefits of targeting one keyword phrase per page vs. multiple keywords per page
What are the benefits of optimizing a page for one keyword phrase versus a group of similar keywords, like this one that Rand posted on another blog entry http://bit.ly/7LzTxY: Ted Baker Ted Baker London Ted Baker Clothing Ted Baker Mens Ted Baker Mens Clothing Ted Baker Mens Collection
On-Page Optimization | | EricVallee340