Amount of Pages Crawled Dropped Significantly
-
I am just wondering if something changed with the SEOMoz crawler. I was always getting 10,000 or near 10,000 pages crawled. After the last two crawls I am ending up around 2500 pages. Has anything changed that I would need to look at it see if I am blocking the crawler or something else?
-
Hi! This is probably a question for the help desk team. If you send them an email at help@seomoz.org and let them know your campaign name, they'll take a look at it and help you figure out what happened.
-
i also want to add that I ran a test crawl and it actually returned 3200 pages, so I am not sure why the site is only doing 2504 for the last two weeks.
-
i did a site update about 6 weeks ago, but was crawled a few times with about 9,000-10,000 pages crawled. It was the last two crawls that it dropped to 2504 pages. I don't use .htaccess, nothing in robots.txt so I am not sure what could be doing it
-
you may be blocking the mozbot. Do you remember the exact date when it started happening? If yes then you can identify and revert the changes. Also check your robots.txt and .htaccess file for any abnormal changes during this time period.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
DA dropped down to 17 from 32
Hi Team, I have observed that DA of website https://www.sonicseats.com/ is dropped down to 17 from 32 after MOZ 2.0 DA Algorithm Update. Though I can see that there is linear improvement observed on Moz. Please help me know in detail about the drop and how can it be improved further as Moz does not give the option to disavow bad back links. Thanks
Moz Pro | | insidewebanalytics1 -
Does the page authority data also considers the on page factors like the presence of keyword in the title,meta text, and keyword frequency ??
The moz difficulty score considers four factors for the top websites. are the on page factors included in the page authority data ?
Moz Pro | | iQuanti0 -
Adjusting SEOmoz Crawling Speed
How do you adjust the SEOmoz crawling speed? SEOmoz tried to crawl 10,000 pages in 3 hours and crashed our MySQL server.
Moz Pro | | cappuccino891 -
Current on-page best practices
Given all the recent talk about over optimization, when was the last time SEOMoz updated the on-page report card tool? Rand wrote an excellent piece on Perfect On-Page Optimization (which is great, and thanks) in summer 2009. Is that still best practice 3 years later (and post-Penguin/Panda)? If not, has the SEOMoz on-page report card tool been updated to reflect current thinking for on-page best-practices? I know the higher level concept is "write for humans, not for bots" but if you can do both (and not create an unreadable seo-frankenpage) then why not? Does getting an "A" grade wreak of over optimization now? Should I use the key phrase at the start of the title, h1, and strong (or bold) elements on a page? Should have an image with file name and alt text equal to (or containing at the start) the key phrase?
Moz Pro | | scanlin2 -
Domain vs. page authority?
hey i've been told that page auth is more important than domain authrity on open site.. why is that?
Moz Pro | | daxvirgo0 -
Page and Domain Authority and other bits
Hi, I am in the process of finding blogs to have a few articles published with a couple of links in each. Articles will all be unique and relevant to the link I drop in and relevant in someway to the reader However I have a few questions. My site is a designer menswear site, so I have picked fashion and sports sites first and foremost to have the articles published. Now, I have a guy who owns about 30 different websites. 2 of them are sports based and about 10 are fashion based. Around $10-$15 an article. I have ran them all through the Open Site Explorer Tool and picked out the best ranked ones. Now my problem is, how do I know if its a good site to not only list an article on, but to pay for it as well. The sites page ranks are around the 30-45 range, the domains are around the 35-45 range. What is a good range to have? I know the higher the better but is 30-45 good enough to pay for? (I don't mind paying the $10 each (£7 my money) for each one) Also as he is quoted me in dollars, I assume there all USA based, so majority of users are USA based. Well I am UK based and only ship to the UK. Will this matter as much if I am trying to gain backlinks? Obviously a UK based site, would be ideal, but is it a case of getting more external links on the web for Google to find, as long as they are relevant to the user? Any help would be great. Thanks Will
Moz Pro | | WillBlackburn0 -
Page Authority vs Domain Authority
I'm hoping this is a quick and simple question. Sometimes when using the open site explorer or the keyword difficulty tool i find that some pages have a higher "page authority" than the "domain authority." That seemed odd to me, is there a correlation between the two? You would think by default as you grow your page authority the domain authority increases as well? So i guess my question would be, what happens on a page to give it a higher page authority than domain authority?
Moz Pro | | kchandler0
