Google crawl rate almost zero since re-launch, organic search up 50% though!
-
We're confused as to why Google's crawl of our site has dropped hugely since our new site went live. The URLs of almost all pages changed, and were 301d to the new site. About 20% of our pages were blocked by robots.txt for the re-launch. The re-launch has been great for organic search, with hits up about 50%. Yet our new content is taking a lot longer to get indexed than before. Our KB downloaded a day according to webmaster tools are well down, as is time spent downloading a page. Any ideas as to why this is?i7hwX.png
-
Hi David,
First, my preference would always be to use 301 redirect to send pages that no longer exist to a new URL.
Using a 301 passes link juice and signals search engines that the old page should be de-indexed and the new page indexed in its place.
Barring some issue with your rel=canonical tags, which would be the first place to look, I would check other things that can send signals to the search engines about freshness:
- meta refresh tags
- update frequency reported in your sitemap
- custom crawl rate setting in Webmaster Tools
Then of course, there is an analysis of the differences between your old design and the new...have you made a significant change to your menu(s) and/or internal linking structure which has made it harder for crawlers to follow links to all pages in your site?
Hope that helps,
Sha
-
It's the same domain but we've rebuilt the website and improved the url and content structure.
We've consolidated urls (e.g. redirecting to canonical url) and published the update xml sitemap.
Pages are crawled correctly, we just can't figure out why the graph changed like that.
-
Hi David,
Just a few questions to help understand your situation:
- When you say "new site" do you mean that you have changed to a new domain?
If it is a new domain:
- Did you 301 individual pages to relevant pages on the new domain, or do you have a catch-all sending all traffic to home?
- Is the chart you uploaded from the new Webmaster Tools account (for the new domain)?
- Have you created and uploaded xml sitemap(s) for the new site?
If it is not a new domain:
- Have you created and uploaded xml sitemap(s) for the new version of the site?
- Have you checked the crawl rate setting in the Webmaster Tools account in case it has been manually altered?
These answers will give us a clearer idea of where to start.
Thanks,
Sha
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How long does Google takes to re-index title tags?
Hi, We have carried out changes in our website title tags. However, when I search for these pages on Google, I still see the old title tags in the search results. Is there any way to speed this process up? Thanks
Technical SEO | | Kilgray0 -
Image Search
Hello Community, I have been reading and researching about image search and trying to find patterns within the results but unfortunately I could not get to a conclusion on 2 matters. Hopefully this community would have the answers I am searching for. 1) Watermarked Images (To remove or not to remove watermark from photos) I see a lot of confusion on this subject and am pretty much confused myself. Although it might be true that watermarked photos do not cause a punishment, it sure does not seem to help. At least in my industry and on a bunch of different random queries I have made, watermarked images are hard to come by on Google's images results. Usually the first results do not have any watermarks. I have read online that Google takes into account user behavior and most users prefer images with no watermark. But again, it is something "I have read online" so I don't have any proof. I would love to have further clarification and, if possible, a definite guide on how to improve my image results. 2) Multiple nested folders (Folder depth) Due to speed concerns our tech guys are using 1 image per folder and created a convoluted folder structure where the photos are actually 9 levels deep. Most of our competition and many small Wordpress blogs outrank us on Google images and on ALL INSTANCES I have checked, their photos are 3, 4 or 5 levels deep. Never inside 9 nested folders.
Technical SEO | | Koki.Mourao
So... A) Should I consider removing the watermark - which is not that intrusive but is visible?
B) Should I try to simplify the folder structure for my photos? Thank you0 -
Google Search Results Display URL
Our urls show as www.domain.com/getproduct.aspx?productid=48376 (url #1) in Google search results. When you click on the link and go to the site the URL is www.domain.com/product-name.aspx (url #2) I checked in Google Webmaster Tools (Fetch as Google) and there is a 302 redirect from url #1 to url #2. It also shows a Set-Cookie value, ASP.NET_SessionID= If we make it a 301 redirect instead, will the url displayed in Google search results be the url #2? We need to get rid of the Set-Cookie for crawlers correct?
Technical SEO | | Guy_Huyett0 -
How to optimize for different google seach center (google.de, google.ch) ?
We all use Deutsch language and (.com) domains for the sites. I ranked well in google.com ,but not so well in google.de , google.ch , my competitors ranked much better in google.de,google.ch. I checked most of their outbound-links, but get few information. Links from (.DE) domains or links from sites located in German help the rank for special google seach center ? (google.de, google.ch) . Or some other factors i missed? please help.
Technical SEO | | sunvary0 -
Google Places Reviews
Has anyone had any delays on Google+ reviews to show up? We have multiple clients who have not received a new review in over two months. These are good accounts with good Zagat scores with 15+ good reviews from real customers. Our clients have asked their clients and have confirmed that there has been reviews left recently. However no new reviews have shown up in the past 60+ days.
Technical SEO | | CaseyKluver0 -
Googlebot Crawl Rate causing site slowdown
I am hearing from my IT department that Googlebot is causing as massive slowdown/crash our site. We get 3.5 to 4 million pageviews a month and add 70-100 new articles on the website each day. We provide daily stock research and marke analysis, so its all high quality relevant content. Here are the crawl stats from WMT: http://imgur.com/dyIbf I have not worked with a lot of high volume high traffic sites before, but these crawl stats do not seem to be out of line. My team is getting pressure from the sysadmins to slow down the crawl rate, or block some or all of the site from GoogleBot. Do these crawl stats seem in line with sites? Would slowing down crawl rates have a big effect on rankings? Thanks
Technical SEO | | SuperMikeLewis0 -
Explain this search result
Hi folks, I came across a strange search result. Search on Google Australia for "income portfolio". http://www.google.com.au/search?sourceid=chrome&ie=UTF-8&q=income+portfolio See the first result? It's a login page. How is that search result showing? And in position #1! Where is it getting its title and descriptions tags from? Does Google have a way to somehow see what is behind the login? Appreciate your thought.
Technical SEO | | scotennis0 -
When is the last time Google crawled my site
How do I tell the last time Google crawled my site. I found out it is not the "Cache" which I had thought it was.
Technical SEO | | digitalops0