Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you. The site has approx 40 sitemaps (one for each main category). They were submitted by myself a little over a year ago, so I'm not convinced it could be related to this? As a sidenote, when i did submit the sitemaps, there was no difference whatsover in crawl rate/pages indexed, which left me feeling a little disappointed!
I have spent the last year removing pages that didn't need to exist (down from 1.5 mill to 110,000). The site has been cleaned up a lot! But nothing substantial recently.
I should also now add that since 9/18 it has gone back down to a more "normal" crawl rate of approx 1500 per day.
I have no idea what caused it! Thanks for your help and advice though, all appreciated!
-
Hi Silkstream!
One of the only thing I can think of is if someone submitted the sitemap.xml multiple times. This could definitely be a reason that you are see such a drastic increase—literally overnight—in the number of pages being indexed.
I would also check your robots.txt file to see if anything has changed. Is it now allowing access to pages that were previously blocked?
Hope this helps!
-
Did you get an influx of new backlinks? Did you create more pages (are there more indexed pages in GSC)? Did you make any on page optimizations?
There appears to be an update rolling out which usually means increased crawler activity.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do I need a separate Google Analytics ID for a subdomain on my website?
Hello Everyone, I would like to know if I need a separate Google Analytics ID for a subdomain on my website? I have a sub-domain within my website, and the software for the site on the subdomain is asking for a Google Analytics ID. Should I be using the same Google Analytics ID as the main domain or should I be creating a new Google Analytics profile or ID? Thanks. Ryan
Reporting & Analytics | | RyanUK0 -
I have conflicting user flow and bounce rates Google Analytics
Has anyone come across this or know how to read this data. In my site content landing page report I have 141 sessions at 0.71% bounce rate = 1 session But then in user flow I have the same page with 123 sessions with a drop off of 117 sessions???
Reporting & Analytics | | Lucas_SOS0 -
Why Is My Traffic Going Up But My Alexa Rating Getting Worse?
I am curious why my Alexa rating has been getting worse over the last few weeks. My organic traffic is up, direct visits and social networking visits are up but my Alexa has been getting worse and worse. Is anyone having the same problem? Any feedback would be helpful. Thanks.
Reporting & Analytics | | Videogamefan0 -
Migrated website but Google Analytics still displays old URL's and none new?!
I migrated a website from a .aspx to a .php and hence had to 301 all the old urls to the new php ones. It's been months after and I'm not seeing any of the php pages showing results but I'm still getting results from the old .aspx pages. Has any one had any experience with this issue or knows what to do? Many thanks,
Reporting & Analytics | | CoGri0 -
Analytics options for website builder
I'm working on a website builder with a few friends and we would like to track traffic statistics for each site built on our platform. We're looking for a solution to dynamically generate analytics for each site published on our platform. I've used Google Analytics for a number of years, but I'm not sure if it's capable of doing something like what we're looking for. Any chance someone could throw me a bone? Thank you in advance!
Reporting & Analytics | | NickMacario0 -
Improving Search Click through Rate
We are having a problem on our website with click through rates. We are getting between 100-150k impressions through search but we are only getting between 500-1000 clicks to the site. What strategies have you used in the past to help improve your click through rates? Thanks!
Reporting & Analytics | | pdangermond2 -
Can underscore blanks trigger home page bounce rates
Buongiorno from 8 degrees C Wetherby UK 🙂 On this site the scrolling banner has been hyperlinked with underscore blank causing new pages to open when a user clicks on a banner. My question is please... "If a user clicks on a banner will Google measure this as a home page bounce" ( I dont think it will i just want to be 100% sure) Click here for illustration:
Reporting & Analytics | | Nightwing
http://i216.photobucket.com/albums/cc53/zymurgy_bucket/home-page-sw-banner_zps0fda6318.jpg Grazie Tanto,
David0 -
My website traffic drop two times
Hi all, on our website www.watchalyzer.com I have unique content that we are writing especially for this online magazine. In last two months our traffic dropped two times. First time on October 20th and after 20 days traffic got back on November 10th. Second time traffic dropped on November 15th and it is still down Does somebody have idea what could be reason for this and how it can be fixed? thanks, Nikola
Reporting & Analytics | | GearyLSF0