Is it possible to block Moz from crawling sites?
-
Hi, is it possible to stop Moz from crawling a site at the server level? Not that I am looking to do this or anything, but here's why I'm asking.
I have been crawling a site that is managed (currently by 2 parties), and I noticed that this week pages crawled went from 80 (last week) to 1 page!! I know, what? See my image attached... and the issues all went to zero "0"....!
So is it possible that someone can't prevent Moz from crawling the site at the server level? I checked the robots.txt file on the site, but nothing there. I'm curious.
-
Thank you both for the responses... I guess it was just a matter of time in my case. It looks like the keywords crawl/report comes in a few hours "after" the initial crawl email from Moz... not sure why, but about 15 minutes after I posted this question, I received the email titled "New Rankings and On-Page Reports Ready" that included the keyword rankings!
Thanks again
-
Here are the most common reasons why only one page is crawled. If this doesn't answer it, send us a note at help@moz.com. Thanks!
-
Ok, here is the helpful resource regarding Roger Mozbot. http://moz.com/help/pro/rogerbot-crawler
This might help!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz and HubSpot SSL - crawl error?
I'm getting an error message when Moz tries to crawl my site, however when I check in Google Search Console, they return no errors. Our site is hosted on HubSpot. Is Moz still having trouble crawling HubSpot sites that have enabled their SSL? I read an article that this should have been corrected in early 2017, but I'm getting an error.
Moz Pro | | jennygriffin0 -
Canonicalisation/ Moz/ WP.
Hi, I've recently put a new site live using Wordpress, and have hooked it up to Moz Analytics. Moz is reporting >1000 pages with 'missing meta description page', and >250 with 'duplicate page content'. The issues seem to be being flagged by pages with '?sort=asc&view=main', ?show=paged&view=main, '?sort=ratedup&show=all&view=list', etc. I have a number of category pages, which users can filter by etc to give them the information in the way they want to see it. I have this ' I haven't got many pages indexed by Google yet, so can't see if they're hitting the same issues as Moz. Any help/ advice appreciated.
Moz Pro | | newstd1000 -
Moz metrics
This discussion is strictly theoretical... I won't hold anyone to their answer. If I have 2 websites that are identical in every way and let's say the domain authority for both is 40, and I 301 redirect one site to the other, what would the DA become? Same question for single pages, both with a PA of 40. If I 301 redirect one page to the other, what does the PA become for the remaining page?
Moz Pro | | AMHC0 -
How do you get your web site recrawled with Moz without waiting for a week?
My initial crawl was screwed up because of a no follow that needed to be removed. I would like Moz to recrawl the site right away so I can find any other errors.
Moz Pro | | Ron_McCabe0 -
Where are my SEO Moz resources
I logged in, changed my password and now I can't find my SEO resources. I need to pull a report quickly, and can't find what I need. Please help ASAP!
Moz Pro | | bcbsm0 -
What do the dates refer to in seo moz reports
question is in the title - new trainee asked me and couldn't actually answer!
Moz Pro | | Highlandgael0 -
Only crawling one page
Hi there, A campaign was crawling fine, but at the last crawl, for some reason, SEOmoz can only crawl one page... any ideas? If I run a custom crawl I still access all of the site's pages.
Moz Pro | | harryholmes0070 -
Crawl completed but no report available for download?
I put 2 crawls in the same day. One came back and delivered a report that I could download. The other one is completed (says so on the page) but there's no way for me to download the report. How do I get a hold of it? Thanks!
Moz Pro | | LiliArancibia0