Is it safe to not have a sitemap if Google is already crawling my site every 5-10 min?
-
I work on a large news site that is constantly being crawled by Google. Googlebot is hitting the homepage every 5-10 minutes. We are in the process of moving to a new CMS which has left our sitemap nonfunctional. Since we are getting crawled so often, I've met resistance from an overwhelmed development team that does not see creating sitemaps as a priority. My question is, are they right? What are some reasons that I can give to support my claim that creating an xml sitemap will improve crawl efficiency and indexing if we are already having new stories appear in Google SERPs within 10-15 minutes of publication? Is there a way to quantify what the difference would be if we added a sitemap?
-
I agree with Robert on all points.
To keep it out of the dev team's overwhelmed hands, just use http://code.google.com/p/googlesitemapgenerator/ or one of the many free generators online to create your sitemaps intermittently.
Maybe 3 months or 6 months down the road the dev team can come up with something when they're less crushed from the site move and you can have them do something similar to Google XML sitemaps plugin for Wordpress which updates the sitemap everytime you add new content. Until then, submitting the freely generated ones should give Google at least a little heads up and feel like you're doing the right thing.
-
As to your 1, I would agree and suggest that it is important on a couple of SEO levels. If you have just updated a story and by virtue of that you have freshened the content. I would want that indexed quickly to move it up if at all possible. However, if you can tell in GWMT that the site is being indexed a couple of times an hour, I am not sure it strengthens your argument.
As to your 2, I would say yes, but if you did a canonical or a 301 on the previous URL - as you should have - it is irrelevant.
Best,
-
Thanks Robert. As you surmised, our URLs are not changing (thankfully!). Fortunately, for now, our Google News sitemap still works. The only arguments I've come up with so far are:
- Having a sitemap will help SEs recrawl updated stories faster.
- Having a sitemap will help SEs find out when a URL has changed.
In my experience, Google does not index changes to existing pages as quickly as newly published articles. My thinking is that if we supply the changes via sitemap, reindexing speed will improve.
Thoughts?
-
Jon
You state you are a news site and you are moving to a new CMS. Assuming the Domain, URL's are the same, I can understand the dev team resistance. This is from WebMaster tools around news sites (bold is mine):
A Google News Sitemap can help you control which content Google News crawls and can speed up the inclusion of your articles in Google News search results. You're welcome to submit your sitemap in your Webmaster Tools account prior to submitting your site for inclusion in Google News. However, only sitemaps associated with an approved site will be crawled without error by Google News.
So, assuming you are already a Google News approved site, you can most likely move forward without immediately submitting a site map. Call me old fashion, but I still think a site map submission is important. But, again, I do get the dev teams resistance. Hope this at least assists your argument.
One added bit of info, You could use a sitemap generator to take a load off of them. Here is a list of many sitemap generators. Since I am not in the dev shop, I cannot recommend any, but I do use the Screaming Frog Spider (never used their SM Generator) This way the Dev team would have a bit less work.
Hope it helps you out a bit,
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google News Sitemap creating service
Hi All, I am dealing with google news sitemap. My technical guy don't know how to create a site for google news. Do you know which service or company can help me with this? Thanks a lot!
Intermediate & Advanced SEO | | binhlai0 -
Keyword On Page 1 Everywhere but Google (Site Specific)
Website: www.wheelchairparts.com
Intermediate & Advanced SEO | | Mike.Bean
Keyword: wheelchair parts My website is #1 or #2 on almost every search engine besides Google. Google has us bouncing between the bottom of page 2 and top of 3. However we are on page one for "wheelchairparts". I need to get a link building campaign going for this site. I feel it's more difficult for ecommerce websites and nothing seems to fit in with Rand's Mozcon 2016 Link Building talk except hacks. I need to find a flywheel. Either way, my question is what can I do other than link building to get on page 1 of Google for the term "wheelchair parts"? Thanks in advance! - Mike Bean1 -
Mobile First Index: What Could Happen To Sites w Large Desktop but Small Mobile Sites?
I have a question about how Mobile First could affect websites with separate (and smaller) mobile vs desktop sites. Referencing this SE Roundtable article (seorountable dot com /google-mobile-first-index-22953.html), "If you have less content on your mobile version than on your desktop version - Google will probably see the less content mobile version. Google said they are indexing the mobile version first." But Google/ Gary Illyes are also on the record stating the switch to mobile-first should be minimally disruptive. Does "Mobile First" mean that they'll consider desktop URLs "second", or will they actually just completely discount the desktop site in lieu of the mobile one? In other words: will content on your desktop site that does not appear in mobile count in desktop searches? I can't find clear answer anywhere (see also: /jlh-marketing dot com/mobile-first-unanswered-questions/). Obviously the writing is on the wall (and has been for years) that responsive is the way to go moving forward - but just looking for any other viewpoints/feedback here since it can be really expensive for some people to upgrade. I'm basically torn between "okay we gotta upgrade to responsive now" and "well, this may not be as critical as it seems". Sigh... Thanks in advance for any feedback and thoughts. LOL - I selected "there may not be a right answer to this question" when submitting this to the Moz community. 🙂
Intermediate & Advanced SEO | | mirabile0 -
Proper 301 in Place but Old Site Still Indexed In Google
So i have stumbled across an interesting issue with a new SEO client. They just recently launched a new website and implemented a proper 301 redirect strategy at the page level for the new website domain. What is interesting is that the new website is now indexed in Google BUT the old website domain is also still indexed in Google? I even checked the Google Cached date and it shows the new website with a cache date of today. The redirect strategy has been in place for about 30 days. Any thoughts or suggestions on how to get the old domain un-indexed in Google and get all authority passed to the new website?
Intermediate & Advanced SEO | | kchandler0 -
Google's form for "Small sites that should rank better" | Any experiences or results?
Back in August of 2013 Google created a form that allowed people to submit small websites that "should be ranking better in Google". There is more info about it in this article http://www.seroundtable.com/google-small-site-survey-17295.html Has anybody used it? Any experiences or results you can share? *private message if you do not want to share publicly...
Intermediate & Advanced SEO | | GregB1230 -
Google Manual Penalty - Unnatural Links FROM My Site - Where?
Hi Mozzers, I've just received a manual penalty for one of my websites. The penalty is for 'unnatural links from my site which I find disturbing because I can't see that anything really wrong with it. The website is www.lighting-tips.co.uk - its a pretty new blog (only 6-7 posts) and whilst I've allowed guest posting I'm being very careful that the content is relevant and good quality. I'm only allowing 1 - 2 links and very few with proper anchor text so I'm wondering what has been done so wrong that I'm getting this manual penalty? Am I missing something here? Thanks in advance. Aaron
Intermediate & Advanced SEO | | AaronGro0 -
How to remove an entire site from Google?
Hi people, I have a site with around 2.000 urls indexed in google, and 10 subdomains indexed too, which I want to remove entirely, to set up a new web. Which is the best way to do it? Regards!
Intermediate & Advanced SEO | | SeoExpertos0 -
Strange situation - Started over with a new site. WMT showing the links that previously pointed to old site.
I have a client whose site was severely affected by Penguin. A former SEO company had built thousands of horrible anchor texted links on bookmark pages, forums, cheap articles, etc. We decided to start over with a new site rather than try to recover this one. Here is what we did: -We noindexed the old site and blocked search engines via robots.txt -Used the Google URL removal tool to tell it to remove the entire old site from the index -Once the site was completely gone from the index we launched the new site. The new site had the same content as the old other than the home page. We changed most of the info on the home page because it was duplicated in many directory listings. (It's a good site...the content is not overoptimized, but the links pointing to it were bad.) -removed all of the pages from the old site and put up an index page saying essentially, "We've moved" with a nofollowed link to the new site. We've slowly been getting new, good links to the new site. According to ahrefs and majestic SEO we have a handful of new links. OSE has not picked up any as of yet. But, if we go into WMT there are thousands of links pointing to the new site. WMT has picked up the new links and it looks like it has all of the old ones that used to point at the old site despite the fact that there is no redirect. There are no redirects from any pages of the old to the new at all. The new site has a similar name. If the old one was examplekeyword.com, the new one is examplekeywordcity.com. There are redirects from the other TLD's of the same to his (i.e. examplekeywordcity.org, examplekeywordcity.info), etc. but no other redirects exist. The chances that a site previously existed on any of these TLD's is almost none as it is a unique brand name. Can anyone tell me why Google is seeing the links that previously pointed to the old site as now pointing to the new? ADDED: Before I hit the send button I found something interesting. In this article from dejan SEO where someone stole Rand Fishkin's content and ranked for it, they have the following line: "When there are two identical documents on the web, Google will pick the one with higher PageRank and use it in results. It will also forward any links from any perceived ’duplicate’ towards the selected ‘main’ document." This may be what is happening here. And just to complicate things further, it looks like when I set up the new site in GA, the site owner took the GA tracking code and put it on the old page. (The noindexed one that is set up with a nofollowed link to the new one.) I can't see how this could affect things but we're removing it. Confused yet? I'd love to hear your thoughts.
Intermediate & Advanced SEO | | MarieHaynes0