Issue in number of pages crawled
-
i wanted to figure out how our friend Roger Bot works.
On the first crawl of one of my large sites, the number of pages crawled stopped at 10000 (due to the restriction on the pro account). However after a few weeks, the number of pages crawled went down to about 5500. This number seemed to be a more accurate count of the pages on our site.
Today, it seems that Roger Bot has completed another crawl and the number is up to 10000 again.
I know there has been no downtime on our site, and the items that we fixed on our site did not reduce or increase the number of pages we had.
Just making sure there are no known issues with Roger Bot before I look deeper into our site to see if there is an issue.
Thanks!
-
Hey Chirag
That is the point, if the crawler is seeing multiple versions of the same page, you will get a false page count.
If a single page resolves on multiple versions of the URL like...
/pagename
/pagename/
/pagename.html
Then one single page could get reported as three pieces of content.
So, if you have 100 pages, but all pages resolve on say two page names then it would show 200 pages BUT the duplicate content report should allow you to see if this is the case.
Hope that helps.
Marcus -
Hi Marcus,
Thanks for the reply.
Yes the duplicate content report is quite large, but I am not certain why the number of pages crawled fluctuated by over 4000.
the Duplicate content number went down by over 2000 last week, and then went straight back up again. So I am not sure if the crawler missed something, or if there was some other issue going on.
Cheers
-
Hey Chirag
As a first suggestion, I would take a look at the duplicate content report and you may see some pages with multiple page names / urls giving a falsely inflated page count.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
MOZ Pro - Page Grader Queries
Greetings! I started a new position last week and the extremely supportive director promised to give me anything I required to make my job easier. Of course, my first port of call was MOZ Pro. Having never used MOZ Pro before, I've just been getting to grips with it, fixing any pressing issues and giving the whole site a general SEO health check. A few fairly major issues have been flagged, which I'm in the process of fixing, and I'm currently putting our main landing pages through the MOZ Page Grader. After a little bit of tinkering, our iPhone 6 cases page has been graded B for the term 'iPhone 6 cases', but I have a few queries/concerns regarding some of the suggested fixes: **Avoid keyword stuffing in document **- The term 'iphone 6 cases' only appears thrice in the body, so it can't be that? The term, however, appears 23 times in the page's img alt tags. Could this be the issue? This is an ecommerce site that sells iPhone 6 cases, so the img alt tags are bound to contain that keyword. Each img alt tag is unique, so I don't really know what I can do here? "Show details for YouSave iPhone 6 0.6mm Clear Gel Case" is the example of the img alt tag of one of the products on that page, surely I can't remove the words 'iPhone 6 case'? Avoid too many internal links - MOZ suggests keeping the internal links to below 100 or, at a minimum, less than 100 links on the main navigation menu. I haven't counted, but I'd guess that page has more than 100 links, but not too many on the navigation menu. To me, this looks like a standard ecommerce page, with links to products and different pages via the top and bottom menus. Would I improve visibility if I reduced the amount of links by, say, reducing the number of products on the page? We currently have it set to 36, but can easily be reduced. Only One Canonical URL - We've put a fix in place for this issue and are just waiting for it to go live. For some reason rel=canonical tags have been duplicated on the majority of the pages. Like I say, this is being remedied, but I just wondered whether a duplicate tag negatively affect the page's visibility? The tags are identical and just point to the page they're on. I think that's about it for now! Thanks in advance and keep up the good work! Cheers, Lewis (Andrew is the name of the director) UPDATE Now I've sorted the rel=canonical issue, the pages are being graded A but still with the first two suggestions above.
Moz Pro | | PeaSoupDigital0 -
ERROR: Too Many on-page links!
User wise, my product menu @LEDSupply.com is user friendly, but I'm concerned that it might be seen by crawlers as bad because of the TOO MANY ON-PAGE LINKS error I am getting in my moz crawl report. Is it really counting all the links in every drop-down menu? If so, is there are resource on how to fix it????
Moz Pro | | saultienut0 -
I have double-checked the rel canonical is properly employed on our page but the On Page Grader says it's not working?
I have double-checked the rel canonical is properly employed on our page but the On Page Grader says it's not working Here is the URL - http://www.solidconcepts.com/industries/aerospace-parts-manufacturing/ What is wrong with how we are doing things?
Moz Pro | | StratasysDirectManufacturing0 -
Number of backlinks throughout time
From what I understand one important aspect in link building is how those backlinks have been acquired throughout time. So if a website gets a large number of backlinks in a short amount of time, the value of those links are reduced in the eyes of search engines. Is there a tool in SEOMoz (that i can't find) that would show the change of backlinks for my campaigns in the last one year?
Moz Pro | | Heydarian0 -
Concerned About Individual Pages
Okay. I've setup a campaign for www.site.com and given a list of keywords. So after the initial crawl we'll have some results. What I'm looking for tho is how do individual pages on my site rank for the list of keywords given. And then be able to go to a screen in seomoz with data for that particular page with recommendations and stuff like that. Is this what's going to happen or do I need to create a campaign for each url i want to track? If all will work as I'd like in the example above, should I then add the second list of keywords that some other pages should rank for? Will it get to be a big mess or can I relate the keywords to pages in some way? It seems like what I'm looking for is what this program should be... Thanks!
Moz Pro | | martJ0 -
How often does seomoz crawl the site? Can you force a crawl at a specific time ?
How often does seomoz crawl the site? Can you force a crawl at a specific time ?
Moz Pro | | stewbuch18720 -
I've got quite a few "Duplicate Page Title" Errors in my Crawl Diagnostics for my Wordpress Blog
Title says it all, is this an issue? The pages seem to be set up properly with Rel=Canonical so should i just ignore the duplicate page title erros in my Crawl Diagnostics dashboard? Thanks
Moz Pro | | SheffieldMarketing0 -
Anyone having issues with OSE?
Everytime i try to access the tool i get the following error: "Error 310 (net::ERR_TOO_MANY_REDIRECTS): There were too many redirects."
Moz Pro | | Anest0