Massive drop off in Google crawl stats
-
Hi
Could i get a second opinion on the following please. ON a client site we seem to have had a massive drop off in google crawling in the past few weeks, this is linked with a drop in search impressions and a slight reduction in penalty. There are no warning messages in WMT to say the site is in trouble, and it shouldn't be, however cannot get to the bottom of what is going on.
In Feb the Kilobytes downloaded per day was between 2200 and about 3800, all good there. However in the past couple of weeks it has peaked at 62 and most days are not even over 3! Something odd has taken place.
For the same period, the Pages crawled per day has gone from 50 - 100 down to under 3.
At the same time the site speed hasn't changed - it is slow and has always been slow (have advised the client to change this but you know how it is....) Unfortunately I am unable to give the site url out so i understand that may impact on any advice people could offer.
Ive attached some screen shots from WMT below.
Many thanks for any assistance.
-
I use it on daily basis, because it gives me quite a good image of what we might miss with manual checking. I hope it will do the trick for you too.
BTW: good to know article for Screaming Frog: http://www.seerinteractive.com/blog/screaming-frog-guide/
That article might speed up things for you and Screaming Frog (for future use )
-
Thanks, will give screaming frog a go. I own the tool but rarely seem to use it. Hopefully that finds out what is amiss.
-
Hi Carl,
What I'd try to do, is to make some crawls via Screaming Frog to the website, try to analyze how your content is crawled, what could have gone wrong.
Also try to keep a timeline of implementations on the website. (I work at a quite big website, and keeping everything organized for myself helps me analyze issues as yours. I will try to make a copy of my excel timeline and post it here in the following days, if you'd like to )
Another thing that you could do, is to fetch and render the pages. Check for anything out of the ordinary.
These are a few ideas of where I would start digging.
I hope it helps. Keszi
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My posts are ignored by google not sure why
I have been investigating this problem for some time now and there must be a technical problem. My posts seem to be ignored by google. For example, if i put this title of my article in google, other sites come up instead of my website Lincolnshire Caravan Owners Struggle To Compete With Haven https://www.in2town.co.uk/skegness-news/lincolnshire-caravan-owners-struggle-to-compete-with-haven/ I am trying to find out what technical problem I have that is stopping google displaying my post. Can anyone advise me on what tools to use and how to find out what is going wrong
Technical SEO | | headlinesplus0 -
Domain Change -- Drop in Rank
So I had a client that recently decided to change their URL to make it more "searchable"/shorter. Since doing so, I have noticed a significant drop in rankings for a number of the keyword terms we were previously ranking for. Is there a way to push the links to the new URL or is the only solution to revert back to the old URL?
Technical SEO | | maxcarnage0 -
Sudden drop in ranking google.co.uk ranking
Anyone else had any sudden drops in rankings this week? Is there an update going on? One of my primary keywords has dropped from 6th to 49th in the google.co.uk search results. Not in webmaster tools to flag an issue. I have downloaded the links from webmaster and it does look if some content has been scraped and then linked back to us from a large number of sites that we have never sort links from. I have upload the google disavow link tool. Only one keyword appears to be effected not all of them? Any ideas? Thanks
Technical SEO | | highwayfive0 -
Google Search memory
Hi we have had the following statement from a member of our Japan office with regards google displaying search results, would anyone be able to give us a definitive answer on this. Google remembers previous non-mobile related searches For example, we already know that we come up on the first page if you select “kaigai keitai” (mobile phone for use abroad) and “UK” where as we don’t for searches where you replace the UK with the US or other countries. This means that if a customer, for example, does a search just on the UK e.g. using words like UK travel, London, millennium dome, etc. and then does a separate search just using the words “kaigai keitai” that google could show us as a link on the first page. However, if an individual did a search on Paris, France, Eiffel Tower, and then did a search for “kaigai keitai”, our link might not appear on the page. I don’t know if we have tested this already, but Google seems to have a very long “memory” and I could see this kind of aspect of Google resulting in us missing significant business from people going to the US, France, Italy, etc. Any thoughts?
Technical SEO | | -Al-0 -
Is this a google dance?
My website keeps moving up and down in ranking but stays within page 2 to 3. Everyday its at a new position.
Technical SEO | | ragivan0 -
Look of google results
Can anyone tell me why some google results show the main page and then a listing of all subsequent pages (i.e. results for SEOMOZ) while others just show the main page with nothing under it. I have two different sites (one personal the other biz) and they both show their search results differently. Is it something in the site creation or how it is crawled by google? Thanks. bKs3C
Technical SEO | | STF0 -
Should we use Google's crawl delay setting?
We’ve been noticing a huge uptick in Google’s spidering lately, and along with it a notable worsening of render times. Yesterday, for example, Google spidered our site at a rate of 30:1 (google spider vs. organic traffic.) So in other words, for every organic page request, Google hits the site 30 times. Our render times have lengthened to an avg. of 2 seconds (and up to 2.5 seconds). Before this renewed interest Google has taken in us we were seeing closer to one second average render times, and often half of that. A year ago, the ratio of Spider to Organic was between 6:1 and 10:1. Is requesting a crawl-delay from Googlebot a viable option? Our goal would be only to reduce Googlebot traffic, and hopefully improve render times and organic traffic. Thanks, Trisha
Technical SEO | | lzhao0 -
Blocking Google from Crawling Parameters
Hi guys: What is the best way to keep Google from crawling certain urls with parameters? I used the setting in Webmaster Tools, but that doesn't seem to be helping at all. Can I use robots.txt or some other method? Thanks! Some examples are: <colgroup><col width="797"></colgroup> www.mayer-johnson.com/category/assistive-technology?manufacturer=179 www.mayer-johnson.com/category/assistive-technology?manufacturer=226 www.mayer-johnson.com/category/assistive-technology?manufacturer=227 <colgroup><col width="797"></colgroup> www.mayer-johnson.com/category/english-language-learners?condition=212 www.mayer-johnson.com/category/english-language-learners?condition=213 www.mayer-johnson.com/category/english-language-learners?condition=214 <colgroup><col width="797"></colgroup>
Technical SEO | | DanaDV
| www.mayer-johnson.com/category/english-language-learners?roles=164 |
| www.mayer-johnson.com/category/english-language-learners?roles=165 |
| www.mayer-johnson.com/category/english-language-learners?roles=197 | | |0