What would cause a drastic drop in pages crawled per day?
-
The site didn't go down.
There were no drop in rankings, or traffic.
But we went from averaging 150,000 pages crawled per day, to ~1000 pages crawled per day.
We're now back up to ~100,000 crawled per day, but we went more than a week with only 1000 pages being crawled daily.
The question is, what could cause this drastic (but temporary) reduction in pages crawled?
-
I wish that were the case, but the site wasn't down.
I looked into the errors, they were redirecting to a subdomain that no longer exists.
-
So several times in one month the entire site couldn't be reached. That's pretty significant. Personally I don't have any clients with that many down-times so can only assume that's the cause or at least a partial cause. And more important, a red flag that would prompt me to find a better hosting provider if it were my site.
-
The drop happened March 28th.
There was a "domain name not found" on march 30th (two more on the 22nd, 18th, 12th, and 10th)
-
There could be several factors. When did it occur? Did you see any other crawl errors reported? And unfortunately, the other unknown comes from the fact that Google's own system is both far from perfect and sometimes crawl volume is affected by their own system.
Unless I see crawl errors or an increase in pages not found during or leading up to that period, or more important, see a corresponding significant drop in organic traffic, personally I just chalk it up to the complexity of the web.
-
Hi Alan!
There were no spikes in kb per day or time spent downloading a page.
-
Fatwallet
Have you checked Google Webmaster Tools for crawl errors and other metrics? I had a client recently who had a severe slowdown in their server network which showed up on page crawl speed time as a huge spike - pages loading five times slower than normal. They subsequently had a dip in pages crawled due to the bottleneck.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When serving a 410 for page gone, should I serve an error page?
I'm removing a bunch of old & rubbish pages and was going to serve 410 to tell google they're gone (my understanding is it'll get them out of the index a bit quicker than a 404). I should still serve an error page though, right? Similar to a 404. That doesn't muddy the "gone" message that I'm giving Google? There's no need to 410 and die?
Intermediate & Advanced SEO | | HSDOnline0 -
New site causes massive drop off in ranking, old site restored how long to recover?
Hello, We launched and updated version of our site, mainly design changes and some functionality. 3 days after the launch we vanished from the rankings, previous page one results were now out of the top 100. We have identified some of the issues with the new site and chose to restore the old well ranking site. My question is how long might it take for the ranking to come back, if at all? The drop happened on the third day and the site was restored on the third day. We are now on day 6. Using GWT with have used fetch as Google and resubmitted the site map. Any help would be gladly received. Thanks James
Intermediate & Advanced SEO | | JamesBryant0 -
Different Header on Home Page vs Sub pages
Hello, I am an SEO/PPC manager for a company that does a medical detox. You can see the site in question here: http://opiates.com. My question is, I've never heard of it specifically being a problem to have a different header on the home page of the site than on the subpages, but I rarely see it either. Most sites, if i'm not mistaken, use a consistent header across most of the site. However, a person i'm working for now said that she has had other SEO's look at the site (above) and they always say that it is a big SEO problem to have a different header on the homepage than on the subpages. Any thoughts on this subject? I've never heard of this before. Thanks, Jesse
Intermediate & Advanced SEO | | Waismann0 -
Does Google still don't index Hashtag Links ? No chance to get a Search Result that leads directly to a section of a page? or to one of numeras Hashtag Pages in a single HTML page?
Does Google still don't index Hashtag Links ? No chance to get a Search Result that leads directly to a section of a page? or to one of numeras Hashtag Pages in a single HTML page? If I have 4 or 5 different hashtag link section pages , consolidated into one HTML Page, no chance to get one of the Hashtag Pages to appear as a search result? like, if under one Single Page Travel Guide I have two essential sections: #Attractions #Visa no chance to direct search queries for Visa directly to the Hashtag Link Section of #Visa? Thanks for any help
Intermediate & Advanced SEO | | Muhammad_Jabali0 -
Will SEO cause a drop in the number of impressions?
Hello, I have been a member of the Moz community for a long time. I very seldom ask questions here but this time I really need your help to make sure I will not make mistakes that will negatively affect my site. My site monetizes according to the number of impressions visualized by the users who visit it. I now want to try to optimize it by using all those nice SEO techniques I have learned through Moz. My goal is to make sure that if I use the various SEO strategies, I will still be able to obtain the high number of impressions I get now. If not, I prefer to leave the situation untouched and I will not start optimizing the site for SEO. Please kindly read the situation below and give me a little support to make sure I am doing the right thing. I would like to ask for your professional advice to solve and issue related to duplicate content. Please visit my site: www.chhedonna.it. The sitemap has indexed 21.890 articles, but if I digit the command 'site:www.chedonna.it', I obtain 158.000 results.I believe that the duplicated content has emerged due to three errors I would like to indicate in order for you to tell me whether my interpretation is correct or not: The article writers made a mistake in that although the content of the articles is different, they have employed the same title for all of them. Therefore, you can see 5 articles with similar/duplicated Tag Titles but the content of said articles is different. For example, http://www.chedonna.it/attualita/cronaca-rosagossip/2012/12/03/heidi-klum-senza-trucco-e-parrucco-foto and http://www.chedonna.it/attualita/cronaca-rosagossip/2012/12/03/heidi-klum-senza-trucco-e-parrucco-foto-2/ are different articles (i.e., the content of the articles is different from each other) that have been published using the same titles. If I inserted the 'follow-noindex' tag to 3 of the 4 duplicates, as shown in the example above, would that be a solution? I fear that if I did insert the follow-noindex, I would cause a drop in the number of impressions visualized by my site's users. It is important to point out, in fact, that I monetize the site via the number of impressions I generate. Therefore, it is fundamental that I do not compromise the number of impressions that the site gets, if I try to optimize it for SEO reasons. On the other hand, I believe that the idea to operate via a 'rel canonical' would not be right, considering that the content of every post having the same title is different, that is, the articles are different even if they focus on the same topic. Also, I would not find it beneficial to use a '301 redirect', since the number of duplicated Titles Tags is very consistent. 2. The second error concerns the duplicate content due to the images that have been included in the articles. For instance, http://www.chedonna.it/che-donna-di-mondo/fare-la-valigia/2012/08/06/campeggio-vacanza-in-liberta/attachment/tenda/ http://www.chedonna.it/che-donna-di-mondo/fare-la-valigia/2012/08/06/campeggio-vacanza-in-liberta/attachment/tenda-2/ http://www.chedonna.it/che-miss/2012/12/04/tatuaggi-fiore-di-loto-significato-e-foto/attachment/tatuaggio-fiore-di-loto-2/ http://www.chedonna.it/che-miss/2013/03/15/tatuaggi-fiore-di-loto-significato-e-foto-2/attachment/tatuaggio-fiore-di-loto-2-2/. I could solve this problem by preventing the media indexing. But I fear that this would produce a very high number of '404 error' messages. If such a thing did happen, my site would stop monetizing overall and I cannot allow this to occur, as you can understand. My very important question is the following: if I prevent the indexing of the photos, will I get in return a drop in the number of impressions that my site would normally generate? 3. Duplicate content generated by the indexing of archive subpages. For example: http://www.chedonna.it/tag/angelica-e-ferdinando/ http://www.chedonna.it/tag/angelica-e-ferdinando/page2/ http://wwwchedonna.it/tag/angelica-e-ferdinando/page3/ If I prevented the media indexing, will I get in return a drop in the number of impressions and many 404 errors? Thank you very much for taking the time to help me sort out this very important issue. Cheers, Sal
Intermediate & Advanced SEO | | salvyy0 -
Page is noindex
Hi, We set pages with this and i can see in the view source of the page <meta name="robots" content="noindex"/> We had a new page posted in the site and its indexed by Google but now the new post is visible on a page thats shows partial data which we noindexed as above because its duplicate data and search engines dont have to see it But its still crawling Any ideas?
Intermediate & Advanced SEO | | mtthompsons0 -
Will Creating a Keyword specific Page to replace the Category Section page cause any harm to my website?
I am running a word press install for my blog and recently had 3 of my main keywords set as categories. I recently decided to create a static page for the keywords instead of having the category page showing all the posts within the category, and took it off the navigation bar. I read about setting the categories to use NO index so the search engines can shine more importance on the new pages i created to really replace where the category was showing. Can this have a negative effect on my rankings? http://junkcarsforcashnjcompany.com junk car removal nj is showing the category section, So i placed the no index on it. Will the search engines refresh the data and replace it with the new page I created?
Intermediate & Advanced SEO | | junkcars0 -
Crawl questions
My first website crawl indicating many issues. I corrected the issues, requested another crawl and received the results. After viewing the excel file I have some questions. 1. There are many pages with missing Titles and Meta Descriptions in the Excel file. An example is http://www.terapvp.com/threads/help-us-decide-on-terapvp-com-logo.25/page-2 That page clearly has a meta description and title. It is a forum thread. My forum software does a solid job of always providing those tags. Why would my crawl report not show this information? This occurs on numerous pages. 2. I believe all my canonical URLs are properly set. My crawl report has 3k+ records, largely due to there being 10 records for many pages. These extra records are various sort orders and style differences for the same page i.e. ?direction=asc. My need for a crawl report is to provide actionable data so I can easily make SEO improvements to my site where necessary. These extra records don't provide any benefit. IF the crawl report determined there was not a clear canonical URL, then I could understand. But that is not the case. An example is http://www.terapvp.com/forums/news/ If you look at the source you will clearly see Where is the benefit to including the 10 other records in the Crawl report which show this same page in various sort orders? Am I missing anything? 3. My robots.txt appropriately blocks many pages that I do not wish to be crawled. What is the benefit to including these many pages in the crawl report? Perhaps I am over analyzing this report. I have read many articles on SEO, but now that I have found SEOmoz, I can see I will need to "unlearn what I have learned". Many things such as setting meta keyword tags are clearly not helpful. I wish to focus my energy and I was looking to the crawl report as my starting point. Either I am missing something, or the report design needs improvement.
Intermediate & Advanced SEO | | RyanKent0