Reasons for a sharp decline in pages crawled
-
Hello!
I have a site I've been tracking using Moz since July. The site is mainly stagnant with some on page content updates. Starting the first week of December, Moz crawler diagnostics showed that the number of pages crawled decreased from 300 to 100 in a week.
So did the number of errors through. So crawler issues went from 275 to 50 and total pages crawled went from 190 to 125 in a week and this number has stayed the same for the last 5 weeks.
Are the drops a red flag? Or is it ok since errors decreased also? Has anyone else experienced this and found an issue?
FYI: sitemap exists and is submitted via webmaster tools. GWT shows no crawler errors nor blocked URLs.
-
Google is indexing just over 80 URLs, although about 40% of them are developer test URLs (they lead to live pages of the site though). Nothing in robots.txt. No errors.
The Google bot is still crawling, but it's crawling half the pages. What would make it decrease in page crawls? I'm working if there is a broken link or something on the home page that's pointing away from the site... although it's unlikely, I'll check....
-
If you fixed a problem, such as duplicate content, that would mean that we're showing fewer errors and crawling fewer pages, since that problem is fixed. Might that be the case?
-
How many URLs are indexed in Google if you use site:yourdomain.com Has that figure dropped too?
Have you got anything in your robots.txt that could be blocking?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How often should I update the content on my pages?
I have started dropping on my rankings - due to lack of time after having a baby. I'm still managing to blog but I'm wondering if I update the content on my pages will that help? All my Meta tags and page descriptions were updated over a year ago - do I need to update these too? We were ranking in the top spots for a good few years, but we're slowly falling 😞 Please give me any advice to keep us from falling even further. I have claimed all my listings, and try to add new links once a month. I share my blog to all social sites and work hard to get Google reviews, we have 53 which is higher than any of our competitors. Any other ideas? Have I missed something that Google is looking for nowadays? Many thanks 🙂
Algorithm Updates | | Lauren16890 -
What does it exactly means when Google brings the "brand name" to the beggining of the page title in search results when it was actually given at the end?
We see many times...page titles starts with "brand name: page for etc" where actually "brand name" has been given at the end and keywords at beginning. Why does Google make this change? I noticed this happens when similar title tags are used by multiple websites for high difficulty keywords. Thanks
Algorithm Updates | | vtmoz0 -
Any suggestions why I would rank 1 on google and be on 3rd page for bing/yahoo?
Currently the site I'm working on ranks very well on google rankings but then when we cross reference into yahoo and bing we are basically in the graveyard of keywords. (bottom of 3rd page). Why would that be? Any suggestions or things I can do to fix this or troubleshoot it? Here are some things I can think of that might affect this but not sure. 1. our sitemap hasn't been updated in months and URL changes have been made 2. Onsite for yahoo and bing is different from google? 3. Bing is just terrible in general? 4. Inbound links? This one doesn't make sense though unless the search engines rank links in different ways. All jokes aside I would really appreciate any help as currently the few top ranked keywords we have are about 30% of our organic traffic and would have a huge affect on the company if we were able to rank as we should across all platforms. Thanks!
Algorithm Updates | | JemJemCertified0 -
Google AMP (accelerated mobile pages), can it be used for non-Google news and Ecommerce Websites?
Mozzers, I've been doing a lot of research on Google's new Accelerated Mobile Pages (AMP) https://moz.com/blog/accelerated-mobile-pages-whiteboard-friday. From what I'm seeing, these AMP version websites are only for Google News-worthy websites such as New York Times, Cosmopolitan, and the BuzzFeeds of the world. But what about Ecommerce websites like Ebay or Amazon? Will AMP versions of "scotch tape" via OfficeDepot work in the SERP's on non-Google News cards?
Algorithm Updates | | Shawn1240 -
Canonical Tag on All Pages
This is a new one for me. I have a client that has a canonical tag on almost every page of their site. Even on pages that don't need it. For example on http://www.client.com/examplex they had code: Maybe I have missed something, but is there a reason for this? Does this hurt the ranking of the page?
Algorithm Updates | | smulto0 -
Geolocation: Google only crawls from the US
A question was previously asked about geo-location and specifically if Google crawled from other countries. I could not locate the original question but wanted to share the below information. As of earlier this year Google only crawls from US IP addresses: http://www.youtube.com/watch?v=7paVYBgH0Hw
Algorithm Updates | | RyanKent1 -
Google place page Images
Is there any real difference in uploading an images directly to your google places page or linking an image from another site? I have heard that you get better results if you upload a photo to photo bucket then to insider pages then post that link to your google places page. To me it just seems a bit odd to do things this way. I get that it's suppose to give you more back links however I don't think it would necessarily be relevant or useful for the user. Any thoughts??
Algorithm Updates | | christinarule0 -
Google said that low-quality pages on your site may affect rankings on other parts
One of my sites got hit pretty hard during the latest Google update. It lost about 30-40% of its US traffic and the future does not look bright considering that Google plans a worldwide roll-out. Problem is, my site is a six year old heavy linked, popular Wordpress blog. I do not know why the article believes that it is low quality. The only reason I came up with is the statement that low-quality pages on a site may affect other pages (think it was in the Wired article). If that is so, would you recommend blocking and de-indexing of Wordpress tag, archive and category pages from the Google index? Or would you suggest to wait a bit more before doing something that drastically. Or do you have another idea what I could to do? I invite you to take a look at the site www.ghacks.net
Algorithm Updates | | badabing0