Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you. The site has approx 40 sitemaps (one for each main category). They were submitted by myself a little over a year ago, so I'm not convinced it could be related to this? As a sidenote, when i did submit the sitemaps, there was no difference whatsover in crawl rate/pages indexed, which left me feeling a little disappointed!
I have spent the last year removing pages that didn't need to exist (down from 1.5 mill to 110,000). The site has been cleaned up a lot! But nothing substantial recently.
I should also now add that since 9/18 it has gone back down to a more "normal" crawl rate of approx 1500 per day.
I have no idea what caused it! Thanks for your help and advice though, all appreciated!
-
Hi Silkstream!
One of the only thing I can think of is if someone submitted the sitemap.xml multiple times. This could definitely be a reason that you are see such a drastic increase—literally overnight—in the number of pages being indexed.
I would also check your robots.txt file to see if anything has changed. Is it now allowing access to pages that were previously blocked?
Hope this helps!
-
Did you get an influx of new backlinks? Did you create more pages (are there more indexed pages in GSC)? Did you make any on page optimizations?
There appears to be an update rolling out which usually means increased crawler activity.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My website has been declining in rankings!
Hello everyone, I have been doing the SEO on the following site: www.painters-decorators-london.co.uk for the past few years and the rankings are slowly declining since last year. (not the recent Google update). I can't seem to figure out what exactly is causing this to happen The page authority and domain authority is higher than my competitors. The on page SEO I believe is done appropriately but I'm not sure whats causing this decline. How can I find out exactly using the MOZ Pro account that I have. I don't think I'm totally aware of the important metrics that one can use on this site to determine the problem. Please advise!
Reporting & Analytics | | Ready2Paint0 -
Drop in Bounce Rate in Google Analytics
Hi guys, I have recently seen a large drop in bounce rate (from GA) which seems unnatural for one of our clients website. Since the start of 2018, the bounce rate was consistently between 40-60%, and then saw a random spike, and now for the past two weeks, the bounce rate is below 10%. I was wondering if anyone had any ideas if this is a problem with GA, or the site itself. Site: https://www.zoomocarcredit.com/ Any comments/feedback is greatly appreciated. Thanks in advance! Jack. SnP0Hc4
Reporting & Analytics | | ChemistryMarketing0 -
Google Webmaster Tools, about multiple entries for your website
Hi I have a doubt about Google Webmaster Tools or Central as it is call today. I remember that google recommended to have one profile of your website for each domain structure. Let me try to be more clear one profile for http://www.yoursite.com, an other for http://yoursite.com, an other for https://www.yoursite.com, etc. Then in each of them we uploaded our sitemaps and cross our fingers. Now from my experience always the complete url have better index status from the sitemap. Now my question is, today as Google requested all our websites run under https, so conserving the other profiles is affecting how google index our pages? shall we have to delete the old profiles or is better to maintain them? Thanks. Pablo
Reporting & Analytics | | FWC_SEO0 -
How can I see what Google sees when it crawls my page?
In other words, how can see the text and what not it sees from start to finish on each page. I know there was a site, but I can't remember it.
Reporting & Analytics | | tiffany11030 -
My GWT tells me that verification has failed numerous occasions - will this stop my site being crawled?
I launched www.over50choices.co.uk 6 weeks ago and have had trouble with google indexing and crawling all pages. It tells me 143 submitted & 129 Indexed, but the site has 166 pages? It still shows the old home page image in GWT - which is v annoying! Whilst the site is verified by GA & HTML Tag, it tells me in the Verification section that "reverification failed" on numerous occasions - they seem correspond with when google trys to process the site map. Is this a coincidence ie verification fails when its trying to process the site map, which in turn is leaving me with an out of date site map and therefore not all my pages submitted or crawled? Or will this not effect the googles ability to crawl the site? Your help please. Ash
Reporting & Analytics | | AshShep10 -
Webmaster tools crawl errors
Hi there, iv been tracking my webmaster tools crawl errors for a while now(6 months) and im noticing some pages that are far gone 404 are still poping out on the crawl errors. - that pages have no data for xml linking, and remote linking are from pages that are far gone 404 also. that pages have 404 error page + redirect to homepage, and google still notice them with old cache content. does someone have a clue why is this happening?
Reporting & Analytics | | Or.Shvartz0 -
Website not responding to web request
Hi, I'm attempting to create a campaign, but the website I want to analyse won't allow SEOmoz to crawl the site stating that the site does not allow 'web requests' - can anything be done about this? Thanks, Adam
Reporting & Analytics | | adamgthorndike0 -
What should I expect after completely redoing a website?
Hello Forum, My team and I have completely redone an eCommerce website, which offers new SEO-friendly URLs. We have also taken care to optimize our product and category pages for specific keywords, so as to keep our regular traffic and increase the long-tail. In the old un-optimized website, many of its Google search results simply took users to the homepage, thus the homepage has a lot of external link juice. Our new site is optimized so that users are taken to the page most relevant to their query. For example (we are a yoga site), with the old site a search for "yoga bolsters" would take visitors to the homepage. Now the search results take the users to the yoga bolster page. These new pages don't have the amount of external links as the new site and I've noticed a decrease in our site's average rank in Google Webmaster Tools 1.Given our new site with its new URLs, should this increase be expected? (we have implemented 301 redirects from the old site to the new) 2. What steps would you take to ensure you keep your rankings with the new site? Are they any special tools you recommend? 3. What the best way to increase the ones that have dropped? Page optimization? Link building? Both?
Reporting & Analytics | | pano2