Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you. The site has approx 40 sitemaps (one for each main category). They were submitted by myself a little over a year ago, so I'm not convinced it could be related to this? As a sidenote, when i did submit the sitemaps, there was no difference whatsover in crawl rate/pages indexed, which left me feeling a little disappointed!
I have spent the last year removing pages that didn't need to exist (down from 1.5 mill to 110,000). The site has been cleaned up a lot! But nothing substantial recently.
I should also now add that since 9/18 it has gone back down to a more "normal" crawl rate of approx 1500 per day.
I have no idea what caused it! Thanks for your help and advice though, all appreciated!
-
Hi Silkstream!
One of the only thing I can think of is if someone submitted the sitemap.xml multiple times. This could definitely be a reason that you are see such a drastic increase—literally overnight—in the number of pages being indexed.
I would also check your robots.txt file to see if anything has changed. Is it now allowing access to pages that were previously blocked?
Hope this helps!
-
Did you get an influx of new backlinks? Did you create more pages (are there more indexed pages in GSC)? Did you make any on page optimizations?
There appears to be an update rolling out which usually means increased crawler activity.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hey DA of my website decreased last night what could be the reason?
Hey DA of my website decreased last night what could be the reason?
Reporting & Analytics | | mozmoz90 -
Why few pages have more than 100% bounce rate?
Hello All, For my ecommerce site approx more than 30k products I have. In Google Analytic approx daily for few products approx 10-15 products bounce rate show 300%, 200%, 150%, 140%, 125% how? and what is the solution? Product page daily change. Thanks!
Reporting & Analytics | | Johny123450 -
Fixing Bounce Rate between Domain and Subdomain
Currently, the way our site is set up, our clients generally visit our homepage and then login through a separate page that is a subdomain, or they can read our blog/support articles that are also on separate subdomains. From my understanding, this can be counted as a bounce, and I know this sorta of site structure isn't ideal, but with our current dev resources and dependencies, fixing this isn't going to happen overnight. Regardless, what would be the easiest way to implement this fix witihn the Google Analytics code? EX: If someone visits our site at X.com, and then wants to login at portal.X.com, I don't want to count that as a bounce. Any insight is appreciated! Thanks
Reporting & Analytics | | KathleenDC0 -
Bounce Rate: what is it EXACTLY?
Hi everyone: we all know the term 'Bounce Rate'. I'd like to think i have a good idea of what BR is....but some things are not really clear to me. Time to call in the experts. Question #1: What EXACTLY will stop Google from considering the visit as a bounce? As discussed not too long ago in this topic https://moz.com/community/q/will-this-fix-my-bounce-rate
Reporting & Analytics | | BasKierkels
Ruben wrote: "..what it basically means is that someone clicks on your SERP, and then clicks back to google? But, it doesn't matter if they spent 10 minutes on your page or 10 seconds" Jessica Conflitti wrote a reply in which she basically said that it might be a good idea to have visitors click to a different page OR a PDF-file. That's where my confusion has been for some time now: Clicking on a PDF-document, an image in the page that opens with Fancybox, a link to a different domain? Or can it only be a different URL on the same domain? The way i would expect it to be:
Pages contain the GA-tracking code. So am i right by thinking that Google needs to have the same GA-tracking code to be loaded twice? Because only at that point will they have two datapoints. And only then will they be able to tell that the visitor hasn't left. By clicking a PDF-document - as described by Jessica - you wouldn't load the GA-code twice. So I would expect that clicking a PDF does not make a difference for the BR. Don't get me wrong: i like the article but it is this detail that throws me off. IF Google can read or capture these clicks, what other elements can be used to reduce bounce rate? Clicking on a YouTube-video embedded in the page? I'm asking this because i want to get this right. Question #2: how much weight does BR have on Time on Page, Engagement, etc? We know Google is taking a lot of things into consideration when calculating the value of a URL or domain. So how much should we care for BR if we know the Time on Page is good and a large percentage of people are frequently returning? How about your experiences or knowledge on that? Really looking forward to your replies and help on clearing this topic for me. And perhaps some other readers as well! Bas0 -
Crawl errors for pages that no longer exist
Hey folks, I've been working on a site recently where I took a bunch of old, outdated pages down. In the Google Search Console "Crawl Errors" section, I've started seeing a bunch of "Not Found" errors for those pages. That makes perfect sense. The thing that I'm confused about is that the "Linked From" list only shows a sitemap that I ALSO took down. Alternatively, some of them list other old, removed pages in the "Linked From" list. Is there a reason that Google is trying to inform me that pages/sitemaps that don't exist are somehow still linking to other pages that don't exist? And is this ultimately something I should be concerned about? Thanks!
Reporting & Analytics | | BrianAlpert780 -
On Google Analytics, Pages that were 301 redirected are still being crawled. What's the issue here?
URL that we redirected are being crawled on Google Analytics. Since they dont exist, they have high bounce rates. What can the issue be?
Reporting & Analytics | | prestigeluxuryrentals.com0 -
How do I add subdomain tracking to an existing Google analytics account that was set up to track website only (without the subdomain option)
I know you can track subdomains by just selecting the proper code when you set up the analytics and then create filters for the data in analytics. But how do you add a subdomain for existing analytics website. Is there a way to go back and change to the option to include subdomains and then I assume just replace the tracking code with the new code that Google delivers for this?
Reporting & Analytics | | rhgraves650 -
Why is this website with worse metrics performing better on serps?
Hi! I would like to ask if anyone has any ideas on why the second website in this analysis is getting position 2 on google and website 1 is getting position 5: http://www.opensiteexplorer.org/comparisons?site=bonobetfair.info&comparisons[0]=www.bonobetfair.com&=Compare The metrics indicate that the first website is clearly superior in (almost) all metrics. Why could this be?
Reporting & Analytics | | inmonova0