XML Sitemap Issue or not?
-
Hi Everyone,
I submitted a sitemap within the google webmaster tools and I had a warning message of 38 issues.
Issue: Url blocked by robots.txt.
Description: Sitemap contains urls which are blocked by robots.txt.
Example: the ones that were given were urls that we don't want them to be indexed: Sitemap: www.example.org/author.xml
Value: http://www.example.org/author/admin/
My issue here is that the number of URL indexed is pretty low and I know for a fact that Robot.txt aren't good especially if they block URL that needs to be indexed. Apparently the URLs that are blocked seem to be URLs that we don't to be indexed but it doesn't display all URLs that are blocked.
Do you think i m having a major problem or everything is fine?What should I do? How can I fix it?
FYI: Wordpress is what we use for our website
Thanks
-
Hi Dan
Thanks for your answer. Would you really recommend using the plugin instead of just uploading the xml sitemap directly to the website's root directory? If yes why?
Thanks
-
Lisa
I would honestly switch to the Yoast SEO plugin. It handles the SEO (and robots.txt) a lot better, as well as the XML sitemaps all within that one plugin.
I'd check out my guide for setting up WordPress for SEO on the moz blog.
Most WP robots.txt files will look like this;
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/
And that's it.
You could always just try changing yours to the above setting first,
before switching to Yoast SEO - I bet that would clear up
the sitemap issues.
Hope that helps!
-Dan ```
-
Lisa, try checking manually which URL is not getting indexed in Google. Make sure you do not have any no follows on those pages. If all the pages are connected / linked together, then Google will crawl your whole site eventually, just a matter of time.
-
Hi
when generating sitemap there are 46 URLs detected by xml-sitemaps.com but when adding the sitemap to WMT only 12 get submitted and 5 are indexed which is really kind of worrying me. This might be because of the xml sitemap plugin that I installed. May be something is wrong with my settings(doc attached 1&2)
I am kind of lost especially that SEOmoz hasn't detected any URLs blocked by Robot.txt
It would be great if you could tell me what should I do next ?
Thanks
-
The first question i would ask is how big is the difference. If the difference is a large in the # of pages on your site and the ones indexed by Google, then you have an issue. The blocked pages might be the ones linking to the ones that have not been indexed and causing issues. Try removing the no follow on those pages and then resubmit your sitemap and see if that fixes the issue. Also double check your site map to make sure you have correctly added all the pages in it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Http to https redirection issue
Hi, i have a website with http but now i moved to https. when i apply 301 redirection from http to https & check in semrush it shows unable to connect with https & similar other tool shows & when i remove redirection all other tools working fine but my https version doesn't get indexed in google. can anybosy help what could be the issue?
Technical SEO | | dhananjay.kumar10 -
Magento technical SEO issues
Hi This is lots of questions and don't expect full answers but if anyone can help or put me in touch with some who can that would be great so here are 3 issues we have from some auditing our site Firstly on pages like https://www.tidy-books.co.uk/shop-with-us/sort-by/price/sort-direction/desc so any pages where there is a sortby the cananoical link doesn't seem to be working correctly. So for here it is https://www.tidy-books.co.uk/shop-with-us/sort-by/price/sort-direction/desc"/> but should be https://www.tidy-books.co.uk/shop-with-us"/> secondly with have a lot of duplicate title tags mainly caused from the blog and the above problem see-> http://prntscr.com/b2t9xe but regarding the blog we have an issue where 2 canonical appearing for example this page
Technical SEO | | tidybooks
https://www.tidy-books.co.uk/blog/page/19/ there are 2 canonical links appearing https://www.tidy-books.co.uk/blog/page/19/"/> we want it to be this
https://www.tidy-books.co.uk/blog/"/> Thirdly
Our mobile usability issues have gone up a lot see- > http://prntscr.com/b2tado
I can see what the issue is that this folder https://www.tidy-books.co.uk/skin/frontend/tidybooks/default/images/ was being crawled by google and contains lots of 'index of' pages. I've disallowed directory in robots.txt as shown here -> http://prntscr.com/b2tbc5 is that correct? any help would be great Just to let you know we use magento v1.7 we use SEO suite ultimate extension and we use fishpigs wordpress extension thanks0 -
301 and 200 Status Issues
Hi, Moz has highlighted that we have duplicate page content on our site, displaying the following: http://bmiresearch.com/press 200 status code and http://www.bmiresearch.com/press 200 status code We have setup a 301 redirect rule on http://bmiresearch.com/press to permanently redirect to http://www.bmiresearch.com/press and on Google inspect element network it shows this http://bmiresearch.com/press 301 status code which mean redirect to this URL permanently http://www.bmiresearch.com/press 200 status code Does anyone know why this might be occuring? Is it possible that because Google has index both URL http://www.bmiresearch.com/press and http://bmiresearch.com/press with 200 status code? If so how would we correct this? Thanks
Technical SEO | | carlsutherland0 -
Has anyone had issues with Bazaarvoice and schema.org?
About a year ago we started using Bazaarvoice to get reviews for our products, and it has been great as far as accumulating content, but Google is not taking the schema.org data and displaying it on the SERP. Someone has told me it is because we are offering multiple products, or that our schema.org tags are incorrect but when I compare our code to other travel sites it seems like everyone is doing something different. This is especially annoying since the Google schema markup check says everything is fine. Does anyone have any advice or similar experiences? Thanks.
Technical SEO | | tripcentral0 -
Pagination Issue
Hello Community, We have a pagination issue on a set of pages : http://homengo.com/s/location/ http://homengo.com/s/location/?page=1 and so on ( ?page=2, ?page=3, ...). As you can see in the source code the pagination rel and prev are there. First question : does moz crawler know and recognize pagination ? Second question : if yes then do you know what could be wrong with pagination on these pages ? Thanks
Technical SEO | | seomengo0 -
How do you handle Wordpress sitemaps within your site?
I have a regular site map on my site and I also have a Wordpress site installed within it that we use for blog/news content. I currently have an auto-sitemap generator installed in Wordpress which automatically updates the sitemap and submits it to the search engines each time the blog is updated. The question I have (which I think I know the answer to but I just want to confirm) is do I have to include all of the articles within the blog in the main site's sitemap despite the Wordpress sitemap having them in there already? If I do include the articles in the main website's sitemap, they would also be in the Wordpress sitemap as well, which is redundant. Redundancy is not good, so I just want to make sure.
Technical SEO | | iresqkeith0 -
Shopify duplicate content issue
We recently moved out site to shopify but now have a duplicate content issue as we have the same products in different collections. I have added canonical code to get rid of this but my webmaster tools still shows hundreds of duplicate pages. How can I tell if the code I added is working? How long will it take for google to recognise this and drop the duplicates from their index and is this likely to have a significant impact on SERPS? Our we page is www.devoted2vintage.co.uk. Thanks Paul
Technical SEO | | devoted2vintage1 -
Sitemap.xml showing up in Google Search
Hello when I do a Google search my sitemap.xml shows up for lots of queries. Does anyone have any advise on this? Should I remove url in Google Webmaster? Thanks,
Technical SEO | | Socialdude0