Best Practices for adding Dynamic URL's to XML Sitemap
-
Hi Guys,
I'm working on an ecommerce website with all the product pages using dynamic URL's (we also have a few static pages but there is no issue with them).
The products are updated on the site every couple of hours (because we sell out or the special offer expires) and as a result I keep seeing heaps of 404 errors in Google Webmaster tools and am trying to avoid this (if possible).
I have already created an XML sitemap for the static pages and am now looking at incorporating the dynamic product pages but am not sure what is the best approach.
The URL structure for the products are as follows:
http://www.xyz.com/products/product1-is-really-cool
http://www.xyz.com/products/product2-is-even-cooler
http://www.xyz.com/products/product3-is-the-coolestHere are 2 approaches I was considering:
1. To just include the dynamic product URLS within the same sitemap as the static URLs using just the following http://www.xyz.com/products/ - This is so spiders have access to the folder the products are in and I don't have to create an automated sitemap for all product
OR
2. Create a separate automated sitemap that updates when ever a product is updated and include the change frequency to be hourly - This is so spiders always have as close to be up to date sitemap when they crawl the sitemap
I look forward to hearing your thoughts, opinions, suggestions and/or previous experiences with this.
Thanks heaps,
LW
-
Hi LW
I agree with Mark re archiving products. Although our products don't expire as quickly as yours appear to do I use http://www.xml-sitemaps.com/standalone-google-sitemap-generator.html on a cron job to keep our sitemap fresh.
I also exclude some of our over dynamic URLs using this tool from appearing in the sitemap.
Dean
-
Hi LW,
What system is backing the online store? Are you using a CMS-driven e-commerce solution?
My suggestion would be to create an automated sitemap for the products. Pay careful attention to the priorities you assign and the update frequencies. (Hourly/daily is fine) I definitely think that you'd be spending far too much time on updating a sitemap if you had to do it manually.
This method will result in you having a more accurate sitemap on crawling.
Also, if you are planning on offering the same project in future, it might be an idea not to remove the product altogether, but rather have a page saying "This offer is currently not available" or something along those lines.
Another option might be to have an archive category of products, where all your expired offers can be placed, not available for order. This could allow you to keep your indexed pages, avoid 404s as well as use the product pages to direct new visitors to related/newer products should they see the products in the archive.
Just thinking out loud.
I'd be interested to see the website and the solution that you do eventually implement.
Regards
Mark
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moving wordpress to it's own server
Our company wants to remove wordpress from our current windows OS server at provider 1 and move it to a new server at provider 2. Godaddy handles our DNS. I would like to have it on the same domain without masking. I would like to make a DNS entry on godaddy so that our current server and our new server can use the same URL (ie sellstuff.com). But I only want the DNS to direct traffic to our current server. The goal here is to have the new server using the same URL as the old server so nothing needs to be masked once traffic is redirected with a 301 rule in the htaccess file. But no traffic outside of the 301 rule will end up going to the new server. I would then like to edit the htaccess file on our current server to redirect to the new servers IP address when someone goes to sellstuff.com/blog. Does this make since and is it possible?
Technical SEO | | larsonElectronics0 -
Some URLs in the sitemap not indexed
Our company site has hundreds of thousands of pages. Yet no matter how big or small the total page count, I have found that the "URLs Indexed" in GWMT has never matched "URLS in Sitemap". When we were small and now that we have a LOT more pages, there is always a discrepancy of ~10% or so missing from the index. It's difficult to know which pages are not indexed, but I have found some that I can verify are in the Sitemap.xml file but not at all in the index. Â When I go to GWMT I can "Fetch and Render" missing pages fine - it's not as though it's blocked or inaccessible. Any ideas on why this is? Is this type of discrepancy typical?
Technical SEO | | Mase0 -
301: Dynamic URL to Static Page
I've been going around trying to get this dynamic url to redirect in the .htaccess file. I know I'm missing something but can't figure it out. Code: RewriteEngine on
Technical SEO | | ohlmanngroup
RewriteCond %{QUERY_STRING} ^/dynamic-url.php?id=43$
RewriteRule ^$ http://static/page/url/inserted/here? [R=301,L] Suggestions?0 -
XML Sitemap Issue or not?
Hi Everyone, I submitted a sitemap within the google webmaster tools and I had a warning message of 38 issues. Issue:Â Url blocked by robots.txt. Description:Â Sitemap contains urls which are blocked by robots.txt. Example: the ones that were given were urls that we don't want them to be indexed: Sitemap:Â www.example.org/author.xml Value:Â http://www.example.org/author/admin/ My issue here is that the number of URL indexed is pretty low and I know for a fact that Robot.txt aren't good especially if they block URL that needs to be indexed. Apparently the URLs that are blocked seem to be URLs that we don't to be indexed but it doesn't display all URLs that are blocked. Do you think i m having a major problem or everything is fine?What should I do? How can I fix it? FYI: Wordpress is what we use for our website Thanks
Technical SEO | | Tay19860 -
Do Seomozers recommend sitemaps.xml or not. I'm thoroughly confused now. The more I read, the more conflicted I get
I realize I'm probably opening a can of worms, but here we go. Do you or do you not add a sitemap.xml to a clients site?
Technical SEO | | catherine-2793880 -
What is Google's Penguin effect on SEO?
I want to know about Google's Penguin. Specially, how it works to protect spam links <seo>or other jobs. Â Â </seo> How I can protect this problem. Kind Regards John
Technical SEO | | JohnDooley0 -
What's the max number of links you should ever have on a page?
Our homepage has a few hundred links and our index pages(pages that link to our spintext pages) have about 900 links on them with no content. Our SEO guy said we have to keep the links under 1000 but I wanted to see what you guys think.
Technical SEO | | upper2bits0 -
How do I 301 url's with numbers in them?
I have a number of 404 error pages showing in webmaster tools and some of the url's have numbers, % symbols, and some are pdf's.  My usual 301 redirect in my htaccess file does NOT redirect these pages where the url's have special characters. What am I doing wrong?
Technical SEO | | BradBorst0