What would cause a drastic drop in pages crawled per day?
-
The site didn't go down.
There were no drop in rankings, or traffic.
But we went from averaging 150,000 pages crawled per day, to ~1000 pages crawled per day.
We're now back up to ~100,000 crawled per day, but we went more than a week with only 1000 pages being crawled daily.
The question is, what could cause this drastic (but temporary) reduction in pages crawled?
-
I wish that were the case, but the site wasn't down.
I looked into the errors, they were redirecting to a subdomain that no longer exists.
-
So several times in one month the entire site couldn't be reached. That's pretty significant. Personally I don't have any clients with that many down-times so can only assume that's the cause or at least a partial cause. And more important, a red flag that would prompt me to find a better hosting provider if it were my site.
-
The drop happened March 28th.
There was a "domain name not found" on march 30th (two more on the 22nd, 18th, 12th, and 10th)
-
There could be several factors. When did it occur? Did you see any other crawl errors reported? And unfortunately, the other unknown comes from the fact that Google's own system is both far from perfect and sometimes crawl volume is affected by their own system.
Unless I see crawl errors or an increase in pages not found during or leading up to that period, or more important, see a corresponding significant drop in organic traffic, personally I just chalk it up to the complexity of the web.
-
Hi Alan!
There were no spikes in kb per day or time spent downloading a page.
-
Fatwallet
Have you checked Google Webmaster Tools for crawl errors and other metrics? I had a client recently who had a severe slowdown in their server network which showed up on page crawl speed time as a huge spike - pages loading five times slower than normal. They subsequently had a dip in pages crawled due to the bottleneck.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Rankings Continue To Drop
Hi there I'm at wits end trying to stop the slow bleed in our rankings to our store URL's that started mid March 2017 and continues through to today. I'd appreciate some pointers and hope this will throw up a challenge to someone out there. Here is the background: 1. We run an e-commerce store on Shopify with a blog. The recent ranking decline has been almost entirely on the store URL's (catalogue and product pages) while at the same time we have seen steady growth in search volumes in the blog - this makes me think we are seeing a Penguin4 penalty of some type, because the impact is confined to the store URLs. 2. We received a linked based manual penalty back in 2014 and this was successfully removed within 3 months. We have quite a large disavow file as a result. 3. Shortly after launch of Penguin 4.0 in Sept/Oct 2016 we saw a really nice boost in traffic and ascribed this to being under a previous Penguin algo penalty, now removed. 4. Come March 2017 we see a small but steady weekly drop in rankings for our store URL's only, this steady drop continues through to today and over time has become significant. Approximately a 50% decline in visitor numbers to store URL's only as of today, since March. All of this despite: a. Initially I thought this was a Panda issue (because it seemed to coincide with Panda releases in March and May) so the entire website has been rewritten (during June and July) with thin content removed across the store and the blog. Remaining content has been given a serious content boost, being very careful to watch for over-optimisation, and for keyword cannibalisation. I think I've got this right. There are also no crawl issues being highlighted by Moz Pro or SEMRush site audits. b. Recently discovered, only last week in fact. A very low domain website, trust score (0 and 0) had been copying our blog articles steadily on a weekly basis, starting Oct 2016 (yes same time as Penguin4) and only caught last week (my fault for missing this). These articles were copied verbatim with all links and so generated nearly 400 spammy backlinks to our store URLs (about 30% of all the links we have). I've had all these articles removed from the spammy site via DMCA so none of those links exist anymore (as of 8/14/17). I've also disavowed this domain with Google. Could these spam links be the issue, and Google is still needing to crawl this site to see the links are no longer there? I'm not sure because my understanding is that Penguin4 would have devalued these links to start? c. A general review of links and anchor text. I've used Moz Pro and SEMRush backlink audit (linked to Google Search Console) and have removed all toxic links by contacting web masters and using Google disavow. This included removing any links that I think are causing over optimised anchor text. After disavow, according to SEMRush, we have no toxic backlinks left and only 50 out of 1200 links with "Money" anchor text. This exercise was completed two days ago when the last disavow file was uploaded. However I don't believe there was an issue here before as toxic links were < 1% of all links and exact match "money" anchor text in the region of 5%. d. One potential problem with our backlinks is that we have quite a few high domain/high trust links to our scholarship page with anchor text "official website". The net result is that our "Other" anchor text category is just over 50% of total links - these are mainly educational institutions with .edu domains. e. A review of internal linking. We had some what I would refer to as SEO links, linking all product and collection pages across the store, through a tagging type system. This was removed two days ago as it was probably unnecessary for user experience. Other than this I have two concerns remaining with our internal linking structure. The first is that we have quite a big static navigation on the left margin of our store collection pages. This is not faceted navigation, but static. The second is that we've internally linked from almost every blog to our "key" money page in the store, however with varied and non-money anchor text. f. There is nothing in Google Search Console indicating a problem, no manual actions, no significant HTML improvements, and Google has indexed over 90% of URL's compared to the sitemap. All broken links have been fixed - there were a lot before but all fixed as of three weeks ago. g. Checking site speed in GA. Speed has remained constant over the period and we have put in some fixes to improve it. Site speed has not got worse and scores average in Googles speed checker. That's about it. It's possible that with the recent changes made with respect to b, c, e and f above I just need to wait a couple more weeks for Google to catch up, and would appreciate thoughts on this. However I'd also like some thoughts on the static navigation on our collection pages, plus importantly on linking from blog articles to mostly a single money page in the store - of all that remains I think this is potentially a problem. Our website is located at www.thekewlshop.com Many thanks for your help. Charles
Intermediate & Advanced SEO | | charlesfitz0 -
Google indexing only 1 page out of 2 similar pages made for different cities
We have created two category pages, in which we are showing products which could be delivered in separate cities. Both pages are related to cake delivery in that city. But out of these two category pages only 1 got indexed in google and other has not. Its been around 1 month but still only Bangalore category page got indexed. We have submitted sitemap and google is not giving any crawl error. We have also submitted for indexing from "Fetch as google" option in webmasters. www.winni.in/c/4/cakes (Indexed - Bangalore page - http://www.winni.in/sitemap/sitemap_blr_cakes.xml) 2. http://www.winni.in/hyderabad/cakes/c/4 (Not indexed - Hyderabad page - http://www.winni.in/sitemap/sitemap_hyd_cakes.xml) I tried searching for "hyderabad site:www.winni.in" in google but there also http://www.winni.in/hyderabad/cakes/c/4 this link is not coming, instead of this only www.winni.in/c/4/cakes is coming. Can anyone please let me know what could be the possible issue with this?
Intermediate & Advanced SEO | | abhihan0 -
Enormous 7 page drop after switching servers and adding load balancers. Thoughts?
Hello Everyone, I'm a longtime Moz user but I had to switch accounts after switching jobs. I was hoping someone might be able to give me some insight on whats going on if possible. Our startup had first page position for our most valuable keyword: "Crowdfunding real estate" for about 6 or 7 months. Once we launched and switched to a production server behind load balancers, we dropped almost overnight to 7th page and we've been there for about a month. We don't have many links yet and some of the ones we DO have are kind of spammy (no idea where they came from and in process of trying to get them removed) but we thought it'd be strange to see that massive drop. We are even pages below a competitor who has NO links and basically zero content on the page. We don't have any notifications in WMT about a manual penalty or anything. I'd really, really appreciate any advice and If anyone has any ideas, the page is at: PatchofLand.com Thanks, Jason
Intermediate & Advanced SEO | | PatchofLand0 -
Merging Sites: Will redirecting the old homepage to an internal page on the new site cause issues?
I've ended up with two sites which have similar content (but not duplicate) and target similar keywords, rather than trying to maintain two sites I would like to merge the sites together. The old site is more of a traditional niche site and targets a particular set of keywords on its homepage, the new site is more of an authority site with a magazine type homepage and targets the same set of keywords from an internal page. My question is: Should I redirect the old site's homepage to the relevant internal page on the new website...
Intermediate & Advanced SEO | | lara_dar
...or should I redirect the old site's homepage to the new site's homepage? (the old site's homepage backlinks are a mixture of partial match keyword anchor text, naked URLs and branded anchor text) I am in two minds (a & b!) (a) Redirecting to the internal page would be great for ranking as there are some decent backlinks and the content is similar (b) But usually when you do a 301 redirect the homepage usually directs to the new homepage and some of the old site's links are related to the domain rather than the keyword (e.g. http://www.site.com) and some people will be looking for the site's homepage. What do you think? Your help is much appreciated (and hope this makes sense...!)0 -
Why does my home page show up in search results instead of my target page for a specific keyword?
I am using Wordpress and am targeting a specific keyword..and am using Yoast SEO if that question comes up.. and I am at 100% as far as what they recommend for on page optimization. The target html page is a "POST" and not a "Page" using Wordpress definitions. Also, I am using this Pinterest style theme here http://pinclone.net/demo/ - which makes the post a sort of "pop-up" - but I started with a different theme and the results below were always the case..so I don't know if that is a factor or not. (I promise .. this is not a clever spammy attempt to promote their theme - in fact parts of it don't even work for me yet so I would not recommend it just yet...) I DO show up on the first page for my keyword.. however.. instead of Google showing the page www.mywebsite.com/this-is-my-targeted-keyword-page.htm Google shows www.mywebsite.com in the results instead. The problem being - if the traffic goes only to my home page.. they will be less likely to stay if they dont find what they want immediately and have to search for it.. Any suggestions would be appreciated!
Intermediate & Advanced SEO | | chunkyvittles0 -
Our keyword rank fell drastically since our last moz crawl. What gives?
Our site was just crawled last night an we had a number of keywords fall 25% or so, from say # 30 to 38. What would cause such a drastic fall? We haven't taken down pages or changed changed page titles, url or content. One thing we are having trouble with is our meta description and meta keywords for each page. Drupal 7 seems to be giving us trouble and the meta information isn't showing up anymore. Would that cause the fall? Any other suggestions?
Intermediate & Advanced SEO | | drudalton1 -
No equivalent page to re-direct to for highly trafficked pages, what should we do?
We have several old pages on our site that we want to get rid of, but we don't want to 404 them since they have decent traffic numbers. Would it be fine to set up a 301 re-direct from all of these pages to our home page? I know the best option is to find an equivalent page to re-direct to, but there isn't a great equivalent.
Intermediate & Advanced SEO | | nicole.healthline0 -
How to enable crawling for dynamic generated search result pages?
I want to enable crawling facility for dynamic generated search result pages which are generating by Magento Solr search. You can view more about it by following URLs. http://code.google.com/p/magento-solr/ http://www.vistastores.com/catalogsearch/result/?q=bamboo+table+lamp
Intermediate & Advanced SEO | | CommercePundit
http://www.vistastores.com/catalogsearch/result/?q=ceramic+table+lamp
http://www.vistastores.com/catalogsearch/result/?q=green+patio+umbrella Right now, Google is not crawling search result page because, I have added following syntax to Robots.txt file. Disallow: /*?q= So, How do I enable crawling of search result pages with best SEO practice? If any other inputs in same direction so, it will help me more to get it done.0