Google crawl rate almost zero since re-launch, organic search up 50% though!

[soulnafein]

We're confused as to why Google's crawl of our site has dropped hugely since our new site went live. The URLs of almost all pages changed, and were 301d to the new site. About 20% of our pages were blocked by robots.txt for the re-launch. The re-launch has been great for organic search, with hits up about 50%. Yet our new content is taking a lot longer to get indexed than before. Our KB downloaded a day according to webmaster tools are well down, as is time spent downloading a page. Any ideas as to why this is?i7hwX.png

[ShaMenz]

Hi David,

First, my preference would always be to use 301 redirect to send pages that no longer exist to a new URL.

Using a 301 passes link juice and signals search engines that the old page should be de-indexed and the new page indexed in its place.

Barring some issue with your rel=canonical tags, which would be the first place to look, I would check other things that can send signals to the search engines about freshness:

• meta refresh tags
• update frequency reported in your sitemap
• custom crawl rate setting in Webmaster Tools

Then of course, there is an analysis of the differences between your old design and the new...have you made a significant change to your menu(s) and/or internal linking structure which has made it harder for crawlers to follow links to all pages in your site?

Hope that helps,

Sha

[soulnafein]

It's the same domain but we've rebuilt the website and improved the url and content structure.

We've consolidated urls (e.g. redirecting to canonical url) and published the update xml sitemap.

Pages are crawled correctly, we just can't figure out why the graph changed like that.

[ShaMenz]

Hi David,

Just a few questions to help understand your situation:

1. When you say "new site" do you mean that you have changed to a new domain?

If it is a new domain:

1. Did you 301 individual pages to relevant pages on the new domain, or do you have a catch-all sending all traffic to home?
2. Is the chart you uploaded from the new Webmaster Tools account (for the new domain)?
3. Have you created and uploaded xml sitemap(s) for the new site?

If it is not a new domain:

1. Have you created and uploaded xml sitemap(s) for the new version of the site?
2. Have you checked the crawl rate setting in the Webmaster Tools account in case it has been manually altered?

These answers will give us a clearer idea of where to start.

Thanks,

Sha