Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you. The site has approx 40 sitemaps (one for each main category). They were submitted by myself a little over a year ago, so I'm not convinced it could be related to this? As a sidenote, when i did submit the sitemaps, there was no difference whatsover in crawl rate/pages indexed, which left me feeling a little disappointed!
I have spent the last year removing pages that didn't need to exist (down from 1.5 mill to 110,000). The site has been cleaned up a lot! But nothing substantial recently.
I should also now add that since 9/18 it has gone back down to a more "normal" crawl rate of approx 1500 per day.
I have no idea what caused it! Thanks for your help and advice though, all appreciated!
-
Hi Silkstream!
One of the only thing I can think of is if someone submitted the sitemap.xml multiple times. This could definitely be a reason that you are see such a drastic increase—literally overnight—in the number of pages being indexed.
I would also check your robots.txt file to see if anything has changed. Is it now allowing access to pages that were previously blocked?
Hope this helps!
-
Did you get an influx of new backlinks? Did you create more pages (are there more indexed pages in GSC)? Did you make any on page optimizations?
There appears to be an update rolling out which usually means increased crawler activity.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hey DA of my website decreased last night what could be the reason?
Hey DA of my website decreased last night what could be the reason?
Reporting & Analytics | | mozmoz90 -
New website server code errors
I launched a new website at www.cheaptubes.com and have recovered my search engine rankings as well after penguin & panda devestation. I'm continuing to improve the site but moz analytics is saying I have 288 medium issues and i see the warning "45% of site pages served 302 redirects during the last crawl". I'm not sure how to fix this. I'm on WP using Yoast SEO so all the 301's I did are 301's not 302's. I do have SSL, could it be Http vs Https?
Reporting & Analytics | | cheaptubes0 -
Bounce Rate % reduce after implementing enhance ecommerce
Hi Guys, My bounce rate is normal 47% to 50% but after implementing enhance ecommerce it is showing 28% dont know what is the reason?
Reporting & Analytics | | varo0 -
Bounce Rate Question - The percent calculated does not add up
Hello All, I'm attempting to see why organic search bounce rate has increased by 5% when compared to last year for a certain section of my website. I am using a custom segment to filter the specific pages I want to look at. Once the custom segment is set, I go to Acquisition - > Channels - > Organic. Then, I click the Landing Pages tab. Because we don't have keyword data anymore the only thing I can look at is the landing pages that contributed to the change in bounce. Finally, I set my date range and compare to the same date range as last year. Once I set the date range I am presented with a list of URLs and the percent change in bounce rate for each URL. This is where I get confused. If you look at the average bounce rate at the top of the column (example 1 attached) it does not add up with the data below it. If you export all of the data to excel, and then do an "Average" function in Excel, the data adds up to 17.29% instead of 35.04% for Sept. 2013. Why does this not add up? Isn't GA calculating the Average? Also, I always notice several URLs with only 1 session per URL. Several of these 1 session URLs have a 100% bounce rate. Since the bounce rate at the top of the column (example1) is a reflection of the average bounce rate, wouldn't these 1 session URLs significantly distort my data? I ultimately just want to see the pages that are contributing to the increased bounce rate when compared to last year. Having a hard time figuring this one out. Thank you all, Dave zMfAGls
Reporting & Analytics | | DaveGuyMan0 -
Anyone heard of or used Hadoop software for SEO/website analytics?
Hey Moz Community, We received an interesting email this morning from a client's marketing manager asking if we knew about or have ever used "Hadoop software" because "[Business owner name] says it’s supposed to help us see how people use our website and more." I've never heard of this software and upon looking around on the web, it says it can do a lot, but my client is a small law firm who average 800-1,000 visitors per month, mainly from their 2 local office cities/markets. Anyone have any thoughts on this or ever use this software? Thanks in advance. Patrick
Reporting & Analytics | | WhiteboardCreations0 -
Is it Possible to request G-WMT Resend emails pertaining to a certain website?
After being added as an administrator of a G-WMT profile, there are no messages from Google in the profile, however the first administrator has a number of messages. Is it possilble to request Google resends these emails so that I get them as well? I have looked around, and assume its not possible, but was wondering if someone had experienced this before and knew of a workaround. Greg
Reporting & Analytics | | AndreVanKets0 -
Distribution of SEOMOZ Ranking Metrics across all Monitored Websites
Hi, I'm looking into building a tool which would incorporate SEOMOZ ranking metrics and I have a few questions with regard to the ranking data which will help me develop it correctly. I have done a few searches, for the information but haven't found an answer to these questions on seomoz - if they exist somewhere then I would be grateful if you could point me in the right direction. 1/. When looking at the values of metrics such as MozRank and DA, is there any information on how this values are distributed amongst the ranked website - EG The mean / median values of DA and MozRank etc of all websites monitored is X & Y . If there were any distribution graphs for these values then even better. 2/. Along the lines of the above - are there any online resources showing similar information for google PR? 3/. What is the calcuation used in the scale of MozRank and DA - ie We have a scale of 1-10 for these values - be we also know the scale in not linear but logrithmic - ie Double the Links of a DA 5 website and you would not end up with a DA 10 website - so what is the calculation which defines the scale. A WBF about google PR by Rand indicated that a value of between 8-9 is used in Google PR - but what about the values used for DA and MozRank etc? Many thanks for your help
Reporting & Analytics | | James770 -
Bounce rates plummeted
In Google Analytics, my average bounce rates plummeted basically overnight. I went from a consistent average daily bounce rate of about 65% to an average daily bounce rate near 5%. My average number of visitors has stayed the same. I don't think there is any significant change I made to my site that may have caused this. Has anyone else had the same problem and know why it happened and how to fix it? Thanks in advance!
Reporting & Analytics | | Ericc220
