Is it possible to block Moz from crawling sites?
-
Hi, is it possible to stop Moz from crawling a site at the server level? Not that I am looking to do this or anything, but here's why I'm asking.
I have been crawling a site that is managed (currently by 2 parties), and I noticed that this week pages crawled went from 80 (last week) to 1 page!! I know, what? See my image attached... and the issues all went to zero "0"....!
So is it possible that someone can't prevent Moz from crawling the site at the server level? I checked the robots.txt file on the site, but nothing there. I'm curious.
-
Thank you both for the responses... I guess it was just a matter of time in my case. It looks like the keywords crawl/report comes in a few hours "after" the initial crawl email from Moz... not sure why, but about 15 minutes after I posted this question, I received the email titled "New Rankings and On-Page Reports Ready" that included the keyword rankings!
Thanks again
-
Here are the most common reasons why only one page is crawled. If this doesn't answer it, send us a note at help@moz.com. Thanks!
-
Ok, here is the helpful resource regarding Roger Mozbot. http://moz.com/help/pro/rogerbot-crawler
This might help!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why my site not crawl?
hi all me allow rogerbot in robots.txt but rogerbot can't crawl my site my site: toska-co.ir
Moz Pro | | jahanidawodi0 -
Malware on Moz?
Hi guys. Today at evening I received few notifications from my anitivirus about malware on Moz. Does anyone have similar issue? q8gWqIo.jpg
Moz Pro | | Dmitriy_S0 -
MOZ Versus HUBSPOT??? If We Discontinue MOZ Subscription and Get HUBSPOT, Anything Lost?
How does Hubspot compare with MOZ? Doe it provide similar features or is the functionality very different? Is Hubspot complementary to MOZ or could it be used as a substitute? If we stopped our MOZ subscription and subscribed to Hubspot would we lose anything? Thanks, Alan
Moz Pro | | Kingalan10 -
Links from Dofollow sites are Nofollow ?
Hii i have made few links from dofollow site through blog commenting method and when i checked my site on Open Site Explorer it shows Nofollow . So my question is links from Dofollow blogs will be nofollow ? There is any benefit of getting links from dofollow site through comments ? Thanks
Moz Pro | | sumit40 -
Indexed sites
Hello I´m a newbie here on seomoz. My question is just simple but i need to be sure about that:Refering to the report which you can create in the "crawl diagnostic summary". Does the CSV export lists all and ONLY the URLS which are in the google index?If not: does a report is available where all and ONLY the URLS are listed which are in google index?Many thanks! Henrik
Moz Pro | | drgoodcat0 -
I am trying to find inbound links for one of my site urls. My question is does SEOMoz able to track all internal links as the Open Site Explorer shows 0 internal links?
It shows 0 internal links when I am pretty sure we have multiple internal links.Should we use absolute urls or relative urls for internal links?
Moz Pro | | SulekhaUSLLC0 -
Conflicting Data with Open Site Explorer & SEO Moz Toolbar
Hi there, Why is my SEOmoz toolbar showing me higher linking root domains than open site explorer? For example, for this site: http://thisisouryear.com/, the linking root domains in SEOMOZ show 1,015 and in open site explorer it says it has 35. I have noticed this for other sites as well. Which one is right? And why are they different?
Moz Pro | | qlkasdjfw1 -
HTTPS site in Open Site Explorer
I'm looking at a site for which the https URL currently ranks in Google. Using a header checker on the http URL I see that it is being 302 redirected to the https version (I have no control or input on this site). In OSE there's no option to specify an https URL as the http part is pre-populated and uneditable. My question is: does OSE treat the https and http version as the same URL? I'm guessing so as the http URL has a lot of domain authority despite not being the "default" URL.
Moz Pro | | Equatorites0