Is it safe to not have a sitemap if Google is already crawling my site every 5-10 min?
-
I work on a large news site that is constantly being crawled by Google. Googlebot is hitting the homepage every 5-10 minutes. We are in the process of moving to a new CMS which has left our sitemap nonfunctional. Since we are getting crawled so often, I've met resistance from an overwhelmed development team that does not see creating sitemaps as a priority. My question is, are they right? What are some reasons that I can give to support my claim that creating an xml sitemap will improve crawl efficiency and indexing if we are already having new stories appear in Google SERPs within 10-15 minutes of publication? Is there a way to quantify what the difference would be if we added a sitemap?
-
I agree with Robert on all points.
To keep it out of the dev team's overwhelmed hands, just use http://code.google.com/p/googlesitemapgenerator/ or one of the many free generators online to create your sitemaps intermittently.
Maybe 3 months or 6 months down the road the dev team can come up with something when they're less crushed from the site move and you can have them do something similar to Google XML sitemaps plugin for Wordpress which updates the sitemap everytime you add new content. Until then, submitting the freely generated ones should give Google at least a little heads up and feel like you're doing the right thing.
-
As to your 1, I would agree and suggest that it is important on a couple of SEO levels. If you have just updated a story and by virtue of that you have freshened the content. I would want that indexed quickly to move it up if at all possible. However, if you can tell in GWMT that the site is being indexed a couple of times an hour, I am not sure it strengthens your argument.
As to your 2, I would say yes, but if you did a canonical or a 301 on the previous URL - as you should have - it is irrelevant.
Best,
-
Thanks Robert. As you surmised, our URLs are not changing (thankfully!). Fortunately, for now, our Google News sitemap still works. The only arguments I've come up with so far are:
- Having a sitemap will help SEs recrawl updated stories faster.
- Having a sitemap will help SEs find out when a URL has changed.
In my experience, Google does not index changes to existing pages as quickly as newly published articles. My thinking is that if we supply the changes via sitemap, reindexing speed will improve.
Thoughts?
-
Jon
You state you are a news site and you are moving to a new CMS. Assuming the Domain, URL's are the same, I can understand the dev team resistance. This is from WebMaster tools around news sites (bold is mine):
A Google News Sitemap can help you control which content Google News crawls and can speed up the inclusion of your articles in Google News search results. You're welcome to submit your sitemap in your Webmaster Tools account prior to submitting your site for inclusion in Google News. However, only sitemaps associated with an approved site will be crawled without error by Google News.
So, assuming you are already a Google News approved site, you can most likely move forward without immediately submitting a site map. Call me old fashion, but I still think a site map submission is important. But, again, I do get the dev teams resistance. Hope this at least assists your argument.
One added bit of info, You could use a sitemap generator to take a load off of them. Here is a list of many sitemap generators. Since I am not in the dev shop, I cannot recommend any, but I do use the Screaming Frog Spider (never used their SM Generator) This way the Dev team would have a bit less work.
Hope it helps you out a bit,
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Old sitemaps after site migration.
Hi, I was wondering if it's safe to remove all the sitemaps from the old site in search console? It's been 3 months since site migration from http://sitea.com (301 redirected) to http://siteb.com. Therefore, can I delete the old sitemap from the http://sitea.com from search console? Thanks.
Intermediate & Advanced SEO | | ggpaul5620 -
Google cache is showing my UK homepage site instead of the US homepage and ranking the UK site in US
Hi There, When I check the cache of the US website (www.us.allsaints.com) Google returns the UK website. This is also reflected in the US Google Search Results when the UK site ranks for our brand name instead of the US site. The homepage has hreflang tags only on the homepage and the domains have been pointed correctly to the right territories via Google Webmaster Console.This has happened before in 26th July 2015 and was wondering if any had any idea why this is happening or if any one has experienced the same issueFDGjldR
Intermediate & Advanced SEO | | adzhass0 -
On 1 of our sites we have our Company name in the H1 on our other site we have the page title in our H1 - does anyone have any advise about the best information to have in the H1, H2 and Page Tile
We have 2 sites that have been set up slightly differently. On 1 site we have the Company name in the H1 and the product name in the page title and H2. On the other site we have the Product name in the H1 and no H2. Does anyone have any advise about the best information to have in the H1 and H2
Intermediate & Advanced SEO | | CostumeD0 -
200 for Site Visitors, 404 for Google (but possibly 200?)
A 2nd question we have about another site we're working with... Currently if a visitor to their site accesses a page that has no content in a section, it shows a message saying that there is no information currently available and the page shows 200 for the user, but shows 404 for Google. They are asking us if it would be better to change the pages to 200's for Google and what impact that might have considering there would be different pages displaying the same 'no information here' message.
Intermediate & Advanced SEO | | Prospector-Plastics0 -
XML Sitemap index within a XML sitemaps index
We have a similar problem to http://www.seomoz.org/q/can-a-xml-sitemap-index-point-to-other-sitemaps-indexes Can a XML sitemap index point to other sitemaps indexes? According to the "Unique Doll Clothing" example on this link, it seems possible http://www.seomoz.org/blog/multiple-xml-sitemaps-increased-indexation-and-traffic Can someone share an XML Sitemap index within a XML sitemaps index example? We are looking for the format to implement the same on our website.
Intermediate & Advanced SEO | | Lakshdeep0 -
Why is Google Still Penalizing My Site?
We got hit pretty hard by Penguin. There were some bad link issues which we've cleared up and we also had a pretty unique situation stemming from about a year ago when we changed the name of the company and created a whole new site with similar content under a different URL. We used the same phone number and address, and left the old site up as it was still performing well. Google didn't care for that so we eventually used 301 redirects to push the link juice from the old site to the new site. That's the background, here's the problem...... We've partially recovered, but there are several keywords that haven't come back anywhere near where they were in Google. We have higher page rank and more links than our competition and are performing in the top 5 for some of our keywords. Other, similar keywords, where we used to be in the top 5, we are now down on page 4 or 5. Our website is www.hudsoncabinetrydesign.com. We build custom cabinetry and furniture in Westchester County, NY just north of NYC. Examples - For "custom built-ins new york" we are number 3 on Google, number 1 on Bing/Yahoo. For "custom kitchen cabinetry ny" we are number 3 on Bing/Yahoo, not in the top 50 on Google. For "custom radiator covers ny" we used to be #1 on Google, are currently #48, currently #2 on Bing/Yahoo. Obviously, we've done something to upset the Google, but we've run out of ideas as to what it could be. Any ideas as to what is going on? Thanks so much for your feedback, Doug B.
Intermediate & Advanced SEO | | doug_b0 -
How to Set Custom Crawl Rate in Google Webmaster Tools?
This is really silly question to set custom crawl rate in Google webmaster tools. Any one can find out that section under setting tab. But, I have confusion to decide number for request per second and second between requests text field. I want to set custom crawl rate for my eCommerce website. I checked my Google webmaster tools and find out as attachment. So, Can I use this facility to improve my crawling? 6233755578_33ce83bb71_b.jpg
Intermediate & Advanced SEO | | CommercePundit0