Is it possible to block Moz from crawling sites?
-
Hi, is it possible to stop Moz from crawling a site at the server level? Not that I am looking to do this or anything, but here's why I'm asking.
I have been crawling a site that is managed (currently by 2 parties), and I noticed that this week pages crawled went from 80 (last week) to 1 page!! I know, what? See my image attached... and the issues all went to zero "0"....!
So is it possible that someone can't prevent Moz from crawling the site at the server level? I checked the robots.txt file on the site, but nothing there. I'm curious.
-
Thank you both for the responses... I guess it was just a matter of time in my case. It looks like the keywords crawl/report comes in a few hours "after" the initial crawl email from Moz... not sure why, but about 15 minutes after I posted this question, I received the email titled "New Rankings and On-Page Reports Ready" that included the keyword rankings!
Thanks again
-
Here are the most common reasons why only one page is crawled. If this doesn't answer it, send us a note at help@moz.com. Thanks!
-
Ok, here is the helpful resource regarding Roger Mozbot. http://moz.com/help/pro/rogerbot-crawler
This might help!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Pro subscription
We have Pro subscription in the name of Convonix and we are not able to perform more than 10 tests on https://analytics.moz.com/pro/link-explorer/ Also, we are not able to see all 10 rows for "Top Followed Links" Request you to help us in this case.
Moz Pro | | Convonix0 -
Why does Moz show 302s that I previously resolved?
So Moz was flagging thousands of 302 errors in the Redirect Issues section. It was due to the URL extensions containing /directory/currency/switch/currency/ I added this to my Robots.txt file as I don't want these indexed. I marked the 302 errors as fixed and after the next crawl they came back and I now get the message: One or more previously fixed issues continue to persist. We have found one or more issues that were marked as fixed previously in the last crawl. Do I just ignore the errors or is there something wrong that I may be doing? Any help would be much appreciated. Thank you.
Moz Pro | | lbagley0 -
MoZ & Other Keyword Tools
Hi There, 1- Does MoZ provide the data on “which keyword searchers are searching in a particular market”, so that I can make a better decision regarding my own keywords?If MoZ does not do this what other best tool is out there in market for this particular purpose? 2- Suppose I decide some keywords for my site and start doing optimization for them. "Does MoZ tell me “how much traffic my those keywords are receiving with the passage of time”. If MoZ does not do this what other best tool is out there in market for this particular purpose? 3- Does Moz give "data on competitors keyword activity also"? If MoZ does not do this what other best tool is out there in market for this particular purpose? Hope somebody with concrete knowledge and experience will enlighten me. Cheers Tanveer
Moz Pro | | Sequelmed0 -
Indexed sites
Hello I´m a newbie here on seomoz. My question is just simple but i need to be sure about that:Refering to the report which you can create in the "crawl diagnostic summary". Does the CSV export lists all and ONLY the URLS which are in the google index?If not: does a report is available where all and ONLY the URLS are listed which are in google index?Many thanks! Henrik
Moz Pro | | drgoodcat0 -
Crawl Diagnostics
Hello, I would appreciate your help on the following issue. During Crawl procedure of e-maximos.com (WP installation) I get a lot of errors of the below mentioned categories: Title Missing or Empty & Missing Meta Description Tag for the URLs: http://e-maximos.com/?like_it=xxxx (i.e. xxxx=1033) Any idea of the reason and possible solution. Thank you in advance George
Moz Pro | | gpapatheodorou0 -
Did moz stop doing webinars?
The last recorded webinar is from april did moz stop doing these? Luckily i have all the moz con videos t go thru (which are awesome by the way-thanks)
Moz Pro | | DavidKonigsberg1 -
Crawl Diagnostics Report
I'm a bit concerned about the results I'm getting from the Crawl Diagnostics Report. I've updated the site with canonical urls to remove duplicate content and when I check the site - it all displays the right values, but the report, which has just finished crawling is still showing a lot of pages as duplicate content. Simple example: http://www.domain.com http://www.domain.com/ Both of them are in the duplicate content section although both have canonical url set as: Does each crawl check the entire site from the beginning or just the pages it didn't have a chance to crawl the last time? This is just one of 333 duplicate content pages, which have canonical url pointing to the right page. Can someone please explain?
Moz Pro | | coremediadesign0 -
The Site Explorer crawl shows errors for files/folders that do not exist.
I'm fairly certain there is ultimately something amiss on our server but the Site Explorer report on my website (www.kpmginstitutes.com) is showing thousands of folders that do not exist. Example: For my "About Us" page (www.kpmginstitutes.com/about-us.aspx), the report shows a link: www.kpmginstitutes.com/rss/industries/404-institute/404-institute/about-us.aspx. We do have "rss", "industries", "404-institute" folders but they are parallel in the architecture, not sequential as indicated in the error url. Has anyone else seen these types of error in your Site Explorer reports?
Moz Pro | | dturkington0