Omitting URLs from XML Sitemap - Bad??
-
Hi all,
We are working on an extremely large retail site with some major duplicate content issues that we are in the process of remedying. The site also does not currently have an XML sitemap.
Would it be advisable to create a small XML sitemap with only the main category pages for the time being, and then after our duplicate content issues are resolved, uploading the complete sitemap? Or should we wait to upload anything until all work is complete down to the product page level and canonicals are in place? Will uploading a incomplete sitemap be fraudulent or misleading in the eyes of the search engines and prompt a penalty, or would having at least the main pages mapped while we continue work be okay?
Please let me know if more info is needed to answer! Thanks in advance!
-
Some good answers here, so I'll just throw in my own 2 cents.
The purpose of a sitemap is to help search engines find pages they might not otherwise find during a regular crawl. Sometimes sitemaps can help pages get indexed faster. Other sitemaps serve special purposes, such as News or Video sitemaps, which can add extra information and help ranking particular types of content.
In reality, many, many sitemaps are incomplete, missing, or flat out wrong. To my knowledge, no search engine will penalize you for this, as they would be penalizing half the web.
The danger of an inaccurate sitemap is that the search engines may chose to ignore it completely. Daune Forrester of Bing has stated that if they find a 1% error rate in your sitemap file, then they will disregard the file. However, no such action is known to exist for incomplete sitemaps.
So I'd say there is little in submitting a sitemap of your truly important page. Unfortunately, this won't stop Google from discovering or crawling your duplicate content issues.
The faster you get these fixed, the better.
-
Hi Thomas,
Definitely comforting to hear that you ran your site with an incomplete sitemap without seeing any negative results. Like I said in my response above, I think we will proceed with the partial sitemap, just to have one on there, and then upload a complete one once we can clean up the site a little more. Thanks for your insights - they were very helpful!
-
Hi Saijo,
Thanks for your recommendations! We do want to place a little more emphasis on our top level and main navigation pages, so I think we will probably proceed with a preliminary sitemap with just those pages for now. Once we get to that point, we'll definitely be needing to use multiple sitemaps and an index - thanks for pointing this out!
-
I ran my site with an incomplete site map for years and didn't seem to have a negative effect. I feel that any site map is better than no sitemap. Sitemaps are such a small part of the SEO equation. What they are most useful for is telling Google what to crawl. Beyond that, I don't believe they have much relevance in passing authority.
-
My Theory On this ( I have no tests to prove this )
If you upload a verified site map thats is essentially telling Google these are the important pages on my site. would you want to risk the importance of the other pages by telling Google you consider only the few important categories as the important ones . They wont drop the other pages completely but MIGHT see them as less important .
I really don't see Google penalizing a site for incomplete sitemaps.
If you have a really large site you might also want to look in to multiple sitemaps : http://googlewebmastercentral.blogspot.com.au/2006/10/multiple-sitemaps-in-same-directory.html
You might also want to look in to the best situations to use rel canonical vs a 301 redirect .
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Indexed, not submitted in sitemap
I have this problem for the site's blog
Technical SEO | | seomozplan196
There is no problem when I check the yoast plugin setting , but some of my blog content is not on the map site but indexed. Did you have such a problem? What is the cause? my website name is missomister1 -
Sitemaps, 404s and URL structure
Hi All! I recently acquired a client and noticed in Search Console over 1300 404s, all starting around late October this year. What's strange is that I can access the pages that are 404ing by cutting and pasting the URLs and via inbound links from other sites. I suspect the issue might have something to do with Sitemaps. The site has 5 Sitemaps, generated by the Yoast plugin. 2 Sitemaps seem to be working (pages being indexed), 3 Sitemaps seem to be not working (pages have warnings, errors and nothing shows up as indexed). The pages listed in the 3 broken sitemaps seem to be the same pages giving 404 errors. I'm wondering if auto URL structure might be the culprit here. For example, one sitemap that works is called newsletter-sitemap.xml, all the URLs listed follow the structure: http://example.com/newsletter/post-title Whereas, one sitemap that doesn't work is called culture-event-sitemap.xml. Here the URLs underneath follow the structure http://example.com/post-title. Could it be that these URLs are not being crawled / found because they don't follow the structure http://example.com/culture-event/post-title? If not, any other ideas? Thank you for reading this long post and helping out a relatively new SEO!
Technical SEO | | DanielFeldman0 -
Xml sitemaps giving 404 errors
We have recently made updates to our xml sitemap and have split them into child sitemaps. Once these were submitted to search console, we received notification that the all of the child sitemaps except 1 produced 404 errors. However, when we view the xml sitemaps in a browser, there are no errors. I have also attempted crawling the child sitemaps with Screaming Frog and received 404 responses there as well. My developer cannot figure out what is causing the errors and I'm hoping someone here can assist. Here is one of the child sitemaps: http://www.sermonspice.com/sitemap-countdowns_paged_1.xml
Technical SEO | | ang0 -
Block bad crawlers
Hi! how are you? I've been working on some of my sites, and noticed that i'm getting lots of crawls by search engines that i'm not intereted in ranking well. My question is the following: do you have a list of 'bad behaved' search engines that take lots of bandwidth and don´t send much/good traffic? If so, do you know how to block them using robots.txt? Thanks for the help! Best wishes, Ariel
Technical SEO | | arielbortz0 -
URL Structure Question
We are building a job board website that will have a decent amount of "career resources" type content and want to make sure we set up our url structure correctly. After researching on Google and here I have an idea how to structure it but would like some insight if we are on the right track. We are using Wordpress for the content part of our website. We will have about 5 content categories (like resume-tips, job-interviews, job-search etc.) The two options we are considering; www.domain.com/career-resources/index.html As content start page www.domain.com/career-resources/resume-tips/index.html category start page www.domain.com/career-resources/resume-tips/top-5-resume-mistakes.html article name is the /career-resources/ folder really needed or can we go something like; www.domain.com/career-resources/index.html As content start page www.domain.com/resume-tips/index.html category start page www.domain.com/resume-tips/top-5-resume-mistakes.html article name Are we on the right track... and is one way better for SEO that the other? Thanks! Shaun
Technical SEO | | aactive0 -
XML Feed
If a site has an xml feed being used by 100 companies to create the content on their site. Will those 100 sites receive any link juice? Is there any way content may be classed as duplicate across these sites? And should the page on the site where the xml feed is coming from have the page indexed first?
Technical SEO | | jazavide0 -
Google News URL Format
Hi, We are currently redesigning our gaming website (www.totallygn.com) and one of our main goals is to get listed by Google News in future. Looking at the Google News URL requirements "The URL for each article must contain a unique number consisting of at least three digits." How does the above affect SEO structure? I was planning on using a format such as www.totallygn.com/xbox-360/360-reviews/fifa-12-review how would this compare to something like? www.totallygn.com/xbox-360/360-reviews/fifa-12-review234 Thanks in advance for your help
Technical SEO | | WalesDragon0 -
URL rewriting from subcategory to category
Hello everybody! I have quite simple question about URL rewriting from subcategory to category, yet I can't find any solution to this problem (due to lack of my deeper apache programming knowledge). Here is my problem/question: we have two website url structures that causes dublicate problems: www.website.lt/language/category/ www.website.lt/language/category/1/ 1 and 2 pages are absolutely same (both also returns 200 OK). What we need is 301 redirect from 2 to 1 without any other deeper categories redirects (like www.website.com/language/category/1/169/ redirecting to .../category/1/ or .../category/). Here goes .htaccess URL rewrite rules: RewriteRule ^([^/]{1,3})/([^/]+)/([^/]+)/([^/]+)/([^/]+)/([^/]+)/$ /index.php?lang=$1&idr=$2&par1=$3&par2=$4&par3=$5&par4=$6&%{QUERY_STRING} [L] RewriteRule ^([^/]{1,3})/([^/]+)/([^/]+)/([^/]+)/([^/]+)/$ /index.php?lang=$1&idr=$2&par1=$3&par2=$4&par3=$5&%{QUERY_STRING} [L] RewriteRule ^([^/]{1,3})/([^/]+)/([^/]+)/([^/]+)/$ /index.php?lang=$1&idr=$2&par1=$3&par2=$4&%{QUERY_STRING} [L] RewriteRule ^([^/]{1,3})/([^/]+)/([^/]+)/$ /index.php?lang=$1&idr=$2&par1=$3&%{QUERY_STRING} [L] RewriteRule ^([^/]{1,3})/([^/]+)/$ /index.php?lang=$1&idr=$2&%{QUERY_STRING} [L] RewriteRule ^([^/]{1,3})/$ /index.php?lang=$1&%{QUERY_STRING} [L] There are other redirects that handles non-www to www and related issues: RedirectMatch 301 ^/lt/$ http://www.domain.lt/ RewriteCond %{HTTP_HOST} ^domain.lt RewriteRule (.*) http://www.domain.lt/$1 [R=301,L] RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_URI} !(.)/$RewriteRule ^(.)$ http://www.domain.lt/$1/ [R=301,L] At this moment we cannot solve this problem with rel canonical (due to our CMS limits). Thanks for your help guys! If You need any other details on our coding, just let me know.
Technical SEO | | jkundrotas0