Drop in number of Pages crawled by Moz crawler
-
What would cause a sudden drop in the number of pages crawled/accessed by the Moz crawler? The site has about 600 pages of content. We have multiple campaigns set up in our Pro account to track different keyword campaigns- but all for the same domain. Some show 600+ pages accessed, while others only access 7 pages for the same domain. What could be causing these issues?
-
If that was how our crawler was accessing your pages before, because there aren't links for them else where, then it could be the issue. It does depend there. There are a lot of aspects to how our crawler works so it is really hard to say.
-
Hi Nick,
Thank you very much.. no actually we did not make any changes to robot.txt, but we have made some changes to sitemap.xml
Could it create a temporary problem?
-
AllaO,
Sorry about the odd crawl activity. I pulled up your campaigns and it looks like something blocked rogerbot from crawling. This could be new redirects or changes in your robots.txt file. If you don't think one of those is the issue get a hold of our support team: https://seomoz.zendesk.com/forums You can submit a ticket from here.
Best of Luck
-
AllaO
Do you have some campaigns set to track at the root and some at the sub domain? That would be the only way I could think of when they are essentially being run simultaneously.
Best
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Pro Tools
Hello I ran into a error while using Moz Pro Tools Crawl Site feature. Stating that my wordpress website could not be crawled. When contacting moz they sent me this screenshot stating the reason for this error is because of the odd ip address highlighted in yellow. Only time I've seen this particular ip-address is during local development. If anyone has any advice on how to fix this or what may have caused this issue. I feel this maybe effecting the site's overall search visibility. ednqnL7
Moz Pro | | willakawillow220 -
Moz is treating my pages as duplicate content but the pages have different content in reality
Attached here is a screenshot of links with duplicate content. Here are some links that is also based on the screenshot http://federalland.ph/construction_updates/paseo-de-roces-as-of-october-2015 http://federalland.ph/construction_updates/sixsenses-residences-tower-2-as-of-october-2015/ http://federalland.ph/construction_updates/sixsenses-residences-tower-3-as-of-october-2015 The links that I have placed here have different content. So I don't why they are treated as duplicates BWWJuvQ
Moz Pro | | clestcruz0 -
Possible Crawling Problem with Screaming Frog and Moz Crawlers
So I'm not sure if what I'm seeing is a problem or not. As of about two weeks ago the Moz crawler has only been able to see www.mysite.com, and none of the links, content, title, ect associated with the page. Essentially the report has one line, what should be the homepage, but it's not able to pull any information from the page but does show a 200 http status code. The report shows nothing blocked by robots or any errors. When I use screaming frog to crawl the site about 75% of the time it just reports one line www.mysite.com with a 200 status code, but again the crawler is not able to actually see the html. The other 25% of the time it works perfectly fine, crawls all pages and sees all meta info and content. There are no errors in Google WMT and everything looks ok there. We have seen a traffic drop the last two weeks but I don't know if this is the reason for it. I can't publicly post the page but if someone has an idea of what might be going on I'd be happy to PM them. Thanks
Moz Pro | | CJ50 -
Why did my rankings drop so fast?
Hi Mozzers, One of my clients have suffered from a sudden ranking loss that started beginning of August. A little background: My client is a big cleaning company that has many franchises nationwide and had very good rankings for the past few months. Generally the busiest time of the year is summer and the 4th quarter is the slowest. Organic Traffic generally drops by 5 to 10% between Q3 and Q4 of each year. 2 things have caught my attention while investigating these ranking losses The latest recent Update from the moz index evaluated the website's DA of 48 when in the summer we had a DA of 56. Never seen such a drop before!! My first question is how accurate is the moz index? How can I make sure this data is accurate? I am very surprised because we have been working hard on getting good quality links for the past year and a half. If the DA is accurate then that could explain the ranking loss Google updating its search engine to the Hummingbird version. Do you guys think this has anything to do with the drop in rankings? The content on the website is not the best but is fairly decent. I am still not sure how the ranking drop could be related to Hummingbird? I have also checked webmaster tools to see if I got hit somehow by penguin and it seems fine. Thanks mozzers
Moz Pro | | Ideas-Money-Art0 -
Lag time between MOZ crawl and report notification?
I did a lot of work to one of my sites last week and eagerly awaited this week's MOZ report to confirm that I had achieved what I was trying to do, but alas I still see the same errors and warnings in the latest report. This was supposedly generated five days AFTER I made the changes, so why are they not apparent in the new report? I am mainly referring to missing metadata, long page titles, duplicate content and duplicate title errors (due to crawl and URL issues). Why would the new crawl not have picked up that these have been corrected? Does it rely on some other crawl having updated (e.g. Google or Bing)?
Moz Pro | | Gavin.Atkinson0 -
Problem crawling a website with age verification page.
Hy every1, Need your help very urgent. I need to crawl a website that first has a page where you need to put your age for verification and after that you are redirected to the website. My problem is that SEOmoz, crawls only that first page, not the whole website. How can I crawl the whole website?, do you need me to upload a link to the website? Thank you very much Catalin
Moz Pro | | catalinmoraru0 -
Duplicate Page Titles and Content
The SeoMoz crawler has found many pages like this on my site with /?Letter=Letter, e.g. http://www.johnsearles.com/metal-art-tiles/?D=A. I believe it is finding multiple caches of a page and identifying them as duplicates. Is there any way to screen out these multiple cache results?
Moz Pro | | johnsearles0 -
Why did SEOMoz only crawl 1 page?
I have multiple campaigns and on a few of them SEOMoz has only crawled one page. I think this may have to do with how I set up the campaign. How do I get SEOMoz to crawl more than one page on these campaigns.
Moz Pro | | HermanAdvertising0