Drop in number of Pages crawled by Moz crawler
-
What would cause a sudden drop in the number of pages crawled/accessed by the Moz crawler? The site has about 600 pages of content. We have multiple campaigns set up in our Pro account to track different keyword campaigns- but all for the same domain. Some show 600+ pages accessed, while others only access 7 pages for the same domain. What could be causing these issues?
-
If that was how our crawler was accessing your pages before, because there aren't links for them else where, then it could be the issue. It does depend there. There are a lot of aspects to how our crawler works so it is really hard to say.
-
Hi Nick,
Thank you very much.. no actually we did not make any changes to robot.txt, but we have made some changes to sitemap.xml
Could it create a temporary problem?
-
AllaO,
Sorry about the odd crawl activity. I pulled up your campaigns and it looks like something blocked rogerbot from crawling. This could be new redirects or changes in your robots.txt file. If you don't think one of those is the issue get a hold of our support team: https://seomoz.zendesk.com/forums You can submit a ticket from here.
Best of Luck
-
AllaO
Do you have some campaigns set to track at the root and some at the sub domain? That would be the only way I could think of when they are essentially being run simultaneously.
Best
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate Page
I just Check Crawl the status error with Duplicate Page Content. As Mentioned Below. Songs.pk | Download free mp3, Hindi Music, Indian Mp3 Songs http://www.getmp3songspk.com Songs.pk | Download free mp3, Hindi Music, Indian Mp3 Songs http://getmp3songspk.com and then i added these lines to my htaccess file RewriteBase /
Moz Pro | | Getmp3songspk
RewriteCond %{HTTP_HOST} !^www.getmp3songspk.com$ [NC]
RewriteRule ^(.*)$ http://www.getmp3songspk.com/$1 [L,R=301] But Still See that error again when i crawl a new test.0 -
Lots of 404s listed in Top Pages by Page Authority
Hi guys, I'm slowly getting to grips with all the aspects of using Moz.
Moz Pro | | giddygrafix
Looking at my link analysis, under the tab labelled Top Pages by Page Authority, there's an awful lot of 404 pages listed. Are these pages which are being linked to (either internally or externally), and should I put a 301 redirect on all the pages listed? I've attached a picture... moz-screen-shot.jpg0 -
Perplexed - Errors increasing, moz rank dropping, conflicting data from other sources. Please Help.
Hi there, I'm in a bit of a bind and could seriously do with some help. For the past month I've been working with a client to resolve errors onsite. In that period my moz rank has plummeted and my errors (dupe title & content) have increased dramatically. The correlation makes sense, the increase - not so much. Here's why, screaming frog is reporting that the majority of duplicate titles have been removed or dealt with. I've also got the assurance of the developer that the suggested changes are being implemented. Unfortunately, I'm not familiar with the CMS - umbraco - and this is the first time I'm working with this particular developer, so I'm not sure how to gauge progress without using moz tools. So, here are my immediate questions: 1. Why would I get different results from Screaming Frog and Rogerbot? 2. Has anyone here been in a similar situation and could they recommend areas to watch for errors creeping in? 3. Is there a way of identifying which errors have occurred when? Narrowing it down to the week each error occurred would be good enough. 4. Is anybody aware of any inherent SEO flaws in umbraco or common SEO mistakes made using the CMS? 5. is there anything I could provide you with to help you help me? 6. Do you have a suggestion? I'm pretty stuck. Thanks in advance.
Moz Pro | | KJDMedia0 -
Is there a easy way to see what pages are crawled?
Hello! Like the questions says... Is there a easy way to see what pages are crawled? I don't mean the ones that have issues, but just the ones that have been crawled? Regards,
Moz Pro | | MattDG0 -
How to resolve Duplicate Content crawl errors for Magento Login Page
I am using the Magento shopping cart, and 99% of my duplicate content errors come from the login page. The URL looks like: http://www.site.com/customer/account/login/referer/aHR0cDovL3d3dy5tbW1zcGVjaW9zYS5jb20vcmV2aWV3L3Byb2R1Y3QvbGlzdC9pZC8xOTYvY2F0ZWdvcnkvNC8jcmV2aWV3LWZvcm0%2C/ Or, the same url but with the long string different from the one above. This link is available at the top of every page in my site, but I have made sure to add "rel=nofollow" as an attribute to the link in every case (it is done easily by modifying the header links template). Is there something else I should be doing? Do I need to try to add canonical to the login page? If so, does anyone know how to do it using XML?
Moz Pro | | kdl01 -
Are you going to update your On-Page too?
With the new over optimisation penalty coming in, I was wondering if you're going to update your online tool to take this into account?
Moz Pro | | photogaz0 -
This Rookie needs help! Duplicate content pages dropped significantly.
So I am pretty new to SEO Moz. I have an e-commerce site and recently did a website redesign. However, not without several mistakes and issues. That said, when SEO Moz did a crawl of my site, the results showed A LOT of Duplicate Content Pages on my site due to my having one item in many variations. It was almost over whelming and because the number of pages was so high, I have been trying to research ways to correct it quickly. The latest crawl from yesterday shows a drastic drop in the number of duplicate content pages and a slight increase in pages with too long page titles (which is fixable). I am embarrassed to give the number of duplicate pages that were showing but, just know, it's been reduced to a third of the amount. I am just wondering if I missed something and should I be happy or concerned? Has there been a change that could have caused this? Thanks for helping this rookie out!
Moz Pro | | AvenueSeo0 -
Why are these pages considered duplicate page content?
A recent crawl diagnostic for a client's website had several new duplicate page content errors. The problem is, I'm not sure where the error comes from since the content in the webpage is different from one another. Here's the pages that SEOMOZ reported to have duplicate page content errors: http://www.imaginet.com.ph/wireless-internet-service-providers-term http://www.imaginet.com.ph/antivirus-term http://www.imaginet.com.ph/berkeley-internet-name-domain http://www.imaginet.com.ph/customer-premises-equipment-term The only thing similar that I see is the headline which says "Glossary Terms Used in this Site" - I hope that the one sentence is the reason for the error. Any input is appreciated as I want to find out the best solution for my client's website errors. Thanks!
Moz Pro | | TheNorthernOffice790