Massive drop off in Google crawl stats
-
Hi
Could i get a second opinion on the following please. ON a client site we seem to have had a massive drop off in google crawling in the past few weeks, this is linked with a drop in search impressions and a slight reduction in penalty. There are no warning messages in WMT to say the site is in trouble, and it shouldn't be, however cannot get to the bottom of what is going on.
In Feb the Kilobytes downloaded per day was between 2200 and about 3800, all good there. However in the past couple of weeks it has peaked at 62 and most days are not even over 3! Something odd has taken place.
For the same period, the Pages crawled per day has gone from 50 - 100 down to under 3.
At the same time the site speed hasn't changed - it is slow and has always been slow (have advised the client to change this but you know how it is....) Unfortunately I am unable to give the site url out so i understand that may impact on any advice people could offer.
Ive attached some screen shots from WMT below.
Many thanks for any assistance.
-
I use it on daily basis, because it gives me quite a good image of what we might miss with manual checking. I hope it will do the trick for you too.
BTW: good to know article for Screaming Frog: http://www.seerinteractive.com/blog/screaming-frog-guide/
That article might speed up things for you and Screaming Frog (for future use )
-
Thanks, will give screaming frog a go. I own the tool but rarely seem to use it. Hopefully that finds out what is amiss.
-
Hi Carl,
What I'd try to do, is to make some crawls via Screaming Frog to the website, try to analyze how your content is crawled, what could have gone wrong.
Also try to keep a timeline of implementations on the website. (I work at a quite big website, and keeping everything organized for myself helps me analyze issues as yours. I will try to make a copy of my excel timeline and post it here in the following days, if you'd like to )
Another thing that you could do, is to fetch and render the pages. Check for anything out of the ordinary.
These are a few ideas of where I would start digging.
I hope it helps. Keszi
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Deindexed homepage by Google
I just noticed that my homepage was de-indexed by google. Any thoughts would be appreciated.
Technical SEO | | Jenny_H0 -
Google + and Google Knoladge Graph
I am trying to get things to match up for the company brand websearch and the Google + page and we have had it for years now The knowledge graph on Google is showing the map, address and name (shown in attached image), but is not linked to a G+ page, as when i click the "Are you the business owner?" its is trying to make me create a new G+ business page. Anyone have any ideas on this? Also does the wiki name have to be exact for it to show? As for phone number would that be coming from the DNS record as that is nowhere in the markup rich snippet or normal markup Thanks in advance LC9cWdG
Technical SEO | | David-McGawn0 -
Homepage is deindexed in Google
Happened sometime on the 12th or 13th of Feb (is there a way to tell exactly besides referring to GA?).
Technical SEO | | Shinosky
I've been on the Google Webmasters Tools forums trying to nail this down - https://productforums.google.com/forum/?utm_medium=email&utm_source=footer#!msg/webmasters/OgpmNCc3IFA/mmtgUilyXUUJ I can only think that Google is viewing this as duplicate content from an internal page for example: http://mudlifeled.com/shop Very frustrating because we were moving up on the first page for some good brand key words and traffic was climbing. Now I've got my hands up and am at a loss to what I can do.0 -
Where does Google pull the date stamp?
We're a news media site with content that has been live for a few years. All of a sudden, Google is showing our content (even though no one has touched the file) with a date stamp of '3 days ago'. Even for content that is years old. I checked the date it was last cached, and it doesn't even match. The URLs were last cached on January 16, but the date stamp says '3 days ago.' From where does Google pull the date stamp? Any ideas?
Technical SEO | | Aggie0 -
Does Google Read Javascript?
I would like to include a list of links in a select type box which I would like google to follow. In order to do this, I will be styling it with the help of javascript, and in turn change the select box into a ul and the options into li's. The li's would each contain a link, but if javascript is disabled it will fallback to a normal css styled select box. My question is would google follow the links made by the javascript? Or would the bot just recognize the select box as a select box and not links. Thanks for any help!
Technical SEO | | BrianJenkins0 -
Pages not indexed by Google
We recently deleted all the nofollow values on our website. (2 weeks ago) The number of pages indexed by google is the same as before? Do you have explanations for this? website : www.probikeshop.fr
Technical SEO | | Probikeshop0 -
Google Fresh Update
Here's a question, if you put the date in your url (often a wordpress format) would that be seen by Google now as 'fresh' content? - part of their new update.
Technical SEO | | pauledwards0 -
Google crawl rate almost zero since re-launch, organic search up 50% though!
We're confused as to why Google's crawl of our site has dropped hugely since our new site went live. The URLs of almost all pages changed, and were 301d to the new site. About 20% of our pages were blocked by robots.txt for the re-launch. The re-launch has been great for organic search, with hits up about 50%. Yet our new content is taking a lot longer to get indexed than before. Our KB downloaded a day according to webmaster tools are well down, as is time spent downloading a page. Any ideas as to why this is?i7hwX.png
Technical SEO | | soulnafein0