Massive drop off in Google crawl stats
-
Hi
Could i get a second opinion on the following please. ON a client site we seem to have had a massive drop off in google crawling in the past few weeks, this is linked with a drop in search impressions and a slight reduction in penalty. There are no warning messages in WMT to say the site is in trouble, and it shouldn't be, however cannot get to the bottom of what is going on.
In Feb the Kilobytes downloaded per day was between 2200 and about 3800, all good there. However in the past couple of weeks it has peaked at 62 and most days are not even over 3! Something odd has taken place.
For the same period, the Pages crawled per day has gone from 50 - 100 down to under 3.
At the same time the site speed hasn't changed - it is slow and has always been slow (have advised the client to change this but you know how it is....) Unfortunately I am unable to give the site url out so i understand that may impact on any advice people could offer.
Ive attached some screen shots from WMT below.
Many thanks for any assistance.
-
I use it on daily basis, because it gives me quite a good image of what we might miss with manual checking. I hope it will do the trick for you too.
BTW: good to know article for Screaming Frog: http://www.seerinteractive.com/blog/screaming-frog-guide/
That article might speed up things for you and Screaming Frog (for future use )
-
Thanks, will give screaming frog a go. I own the tool but rarely seem to use it. Hopefully that finds out what is amiss.
-
Hi Carl,
What I'd try to do, is to make some crawls via Screaming Frog to the website, try to analyze how your content is crawled, what could have gone wrong.
Also try to keep a timeline of implementations on the website. (I work at a quite big website, and keeping everything organized for myself helps me analyze issues as yours. I will try to make a copy of my excel timeline and post it here in the following days, if you'd like to )
Another thing that you could do, is to fetch and render the pages. Check for anything out of the ordinary.
These are a few ideas of where I would start digging.
I hope it helps. Keszi
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website ranking on Google dropping for unknown reason while rankings are improving on Bing. Please help!
one of my websites www.resumeble.com is showing a constant drop in rankings. Earlier the website was ranking for major keywords like resume writing services etc. I used Ahrefs site audit to find issues. According to Ahrefs there was a huge issue of duplicate pages, which is now resolved by proper canonical tag insertion. The site is built on Angular. Fetch report in Google shows perfect code and Sitemap is also perfect. Manual action reporting in webmaster shows no warning. Please suggest what steps should I take to fix this issue.
Technical SEO | | mayyaa40 -
Why Google crawl parameter URLs?
Hi SEO Masters, Google is indexing this parameter URLs - 1- xyz.com/f1/f2/page?jewelry_styles=6165-4188-4184-4192-4180-6109-4191-6110&mode=li_23&p=2&filterable_stone_shapes=4114 2- xyz.com/f1/f2/page?jewelry_styles=6165-4188-4184-4192-4180-4169-4195&mode=li_23&p=2&filterable_stone_shapes=4115&filterable_metal_types=4163 I have handled by Google parameter like this - jewelry_styles= Narrows Let Googlebot decide mode= None Representative URL p= Paginates Let Googlebot decide filterable_stone_shapes= Narrows Let Googlebot decide filterable_metal_types= Narrows Let Googlebot decide and Canonical for both pages - xyz.com/f1/f2/page?p=2 So can you suggest me why Google indexed all related pages with this - xyz.com/f1/f2/page?p=2 But I have no issue with first page - xyz.com/f1/f2/page (with any parameter). Cononical of first page is working perfectly. Thanks
Technical SEO | | Rajesh.Prajapati
Rajesh0 -
Crawl Issues / Partial Fetch Via Google
We recently launched a new site that doesn't have any ads, but in Webmaster Tools under "Fetch as Google" under the rendering of the page I see: Googlebot couldn't get all resources for this page. Here's a list: URL Type Reason Severity https://static.doubleclick.net/instream/ad_status.js Script Blocked Low robots.txt https://googleads.g.doubleclick.net/pagead/id AJAX Blocked Low robots.txt Not sure where that would be coming from as we don't have any ads running on our site? Also, it's stating the the fetch is a "partial" fetch. Any insight is appreciated.
Technical SEO | | vikasnwu0 -
Dropped 20 places!
Hi everyone, I have spent the last 3 weeks using SEO Moz to improve the SEO of my website. I have reduced the amount of crawl errors from 6000 to 600, I have optimized my on-page to A grade for 4 of my main keywords and have been posting keyword optimized blogs posts 3 / 4 times weekly. My reward for this? A drop down to page 4 of Google for my main keyword and a very annoyed boss! Can anyone think where I may have gone wrong? Thanks
Technical SEO | | gaz33421 -
Google Dropping Pages After SEO Clean Up
I have been using SEOmoz to clear errors from a site. There
Technical SEO | | Andy56
were over 10,000 errors to start with. Most of these were duplicate content, duplicate titles and too many links on a page. Most of the duplicate errors have now been
cleared. This has been done in two weeks (down to around 3000 errors now). But instead of improving my rankings, pages that were on the second page of Google have started to drop out of the listings altogether. The pages that are dropping out
are not related to the duplicate problems and get A grades when I run SEOmoz
page reports. Can you clean up too much too quickly or is there likely to be another reason for it?0 -
Sudden MozRank drop?
My site's MozRank dropped abruptly from 3.5 to 3 from September to October. MozRank had been steadily climbing until then and there were no changes made to the site in that time period. What may have happened?
Technical SEO | | mlwilmore0 -
Have a client that migrated their site; went live with noindex/nofollow and for last two SEOMoz crawls only getting one page crawled. In contrast, G.A. is crawling all pages. Just wait?
Client site is 15 + pages. New site had noindex/nofollow removed prior to last two crawls.
Technical SEO | | alankoen1230 -
Websites not being included on google from mobiles?
Hi, Just had a call from a guy saying that google have made a statement saying that it will be stopping people finding websites from mobule devices if they dont have a mobile domain name. Doesn anyone know anything about any Google statements or is this just rubbish?
Technical SEO | | Ant710