Wild fluctuation in number of pages crawled
-
I am seeing huge fluctuations in the number of pages discovered the crawl each week. Some weeks the crawl discovers > 10,000 pages and other weeks I am seeing 4-500.
So, this week for example I was hoping to see some changes reflected for warnings from last weeks report (which discovered > 10,000 pages). However, the entire crawl this week was 448 pages.
The number of pages discovered each week seems to go back and forth between these two extremes. The more accurate count would be nearer the 10,000 mark than the 400 range.
Thanks.
Mark
-
No problem!
Glad to see Cyrus' response!
-
Hi Mark,
I used to troubleshoot these types of problems (mysteries when I worked on the SEOmoz help team.
The best thing to do would be contact the Help Team (help@seomoz.org) and include information both your account, url and campaign. They can take this information and see if there is anything odd about your website, or if there is a bug in the crawling software, or finally if there is some strange quirk of incompatibility causing this behavior.
If you would rather, you can PM me with the info and I can try to troubleshoot it myself, but the Help Team has a few more tools and access to engineers, so they might be the better choice. Either way, let us know if you have any trouble.
-
Thank you for the response. I should have been more clear. It is the weekly SEOMoz crawl that is showing such inconsistent behavior, not Google. Sorry I wasn't more clear.
We have very few (if any) broken links, errors, etc.
Thanks.
Mark
-
Hi there Mark!
We used to have the same issue using Joomla here. It turns out that Google will reduce their crawling if your site has too many errors, broken links, and so on.
We used GWT to look into the 404's then redirected the broken links. Afterwards, we resubmitted the site to be reindexed. A few weeks later -VOILA- all is back to normal and our page freshness stays where it should.
I'd recommend looking at your GWT first, and fixing broken links followed by resubmission to SE's...
Good Luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page Optimization Score is 100%
I have added a page for a particular product, sodium bisulfate, Page Optimization Score is totally green 100%, what should I do, either to sit back and see the results? http://waqaschemicals.com/site/sodium-bisulphate-bisulfate-from-turkey/ I typically do not understand while using the Moz that exactly line of action do I have to follow for to bring the good result for the keyword "sodium bisulfate"? How much does a google takes time to get your 100% things on the first page ? I'm not good in SEO but few knowledge step by step What is the best things to do for good SEO.
Moz Pro | | waqaspuri0 -
Pages with Duplicate Page Content Nov
Moz is showing all many of URL's as duplicate URLs. I put canonical for all the pages but still it showing all as duplicate page. These are URL's https://www.crystalizeonline.com/brands/ravenscroft-crystal/material/non-lead/page/2.html https://www.crystalizeonline.com/brands/ravenscroft-crystal/material/non-lead/page/2/sort-by/price/sort-direction/desc.html https://www.crystalizeonline.com/brands/ravenscroft-crystal/material/non-lead/page/2/sort-by/price/sort-direction/asc.html Their is a lot of pages like this. How can I get rid from all this issues.
Moz Pro | | crystalize0 -
Number of backlinks throughout time
From what I understand one important aspect in link building is how those backlinks have been acquired throughout time. So if a website gets a large number of backlinks in a short amount of time, the value of those links are reduced in the eyes of search engines. Is there a tool in SEOMoz (that i can't find) that would show the change of backlinks for my campaigns in the last one year?
Moz Pro | | Heydarian0 -
No page reports for one of my campaigns
Hi Everyone! so glad to be here, I had hardly visited the forum and been missing out. I have a client who hired me to set up their basic SEO for their website and I see no grades for the pages, at first I don't maybe the website was not being indexed but report shows it crawled 37 pages so what does it mean no grades what so ever, also showing now rankings, I put 30 to check and zilch. thank you.
Moz Pro | | FeuzaReis120 -
What is the difference between the Rank Tracker and the On-page Optimization page?
Both of them track keywords. In the Rank Tracker, you add each keyword manually and you associate it with a URL. For On-page Optimization page, the URLs are generated automatically based on searches and traffic?
Moz Pro | | ehabd0 -
Number of available links limited?
OK, I've been making use of the free LinkScape API (on behalf of a client of mine) and trying to get links (and info on those links) to a specific domain/page/etc. NOTE : I've been using it without any issue in the past, however we are currently facing some weird issues. Let's take this simple query as an example : http://lsapi.seomoz.com/linkscape/links/wikipedia.org?SourceCols=4&TargetCols=4&Sort=page_authority&Scope=page_to_domain What this one supposedly does is to get links to "wikipedia.org", right? I'm reading : The Page_to_* scopes will by default return 25 links per source domain if no limit is specified, so you can see domain diversity. Due to space limitations in our API, a general link query for a given page will return at most 25 pages for every unique domain linking to that page. And I'm saying OK, that's fine. The thing is that (instead of the 1000 links I had been getting before), I'm now getting just 25 links. NOT per... "source domain"... but obviously per "target domain" (= wikipedia.org) - or am I missing something? (well, probably wikipedia suddenly has just about 25 links pointed to it... makes sense! 🙂 ) Please, let me know what's going on with the above, simply because getting just 25 links is close to worthless... Thanks a lot, in advance!
Moz Pro | | drkameleon0 -
SEOMoz only crawling 5 pages of my website
Hello, I've added a new website to my SEOmoz campaign tool. It only crawls 5 pages of the site. I know the site has way more pages then this and also has a blog. Google shows at least 1000 results indexed. Am I doing something wrong? Could it be that the site is preventing a proper crawl? Thanks Bill
Moz Pro | | wparlaman0 -
Only Crawling 1 page?
Hi Guys, Any advice much appreciated on this! Recently set up a new campaign on my dashboard with just 5 keywords. The domain is brammer.co.uk and a quick Google site:brammer.co.uk shows a good amount of indexed pages. However - first seomoz tool crawl has only crawled 1 url!! "Last Crawl Completed: Apr. 12th, 2011 Next Crawl Starts: Apr. 17th, 2011" Any ideas what's stopping the tool crawl anymore of the site?? Cheers in advance.. J
Moz Pro | | lovealbatross0