Is it possible to block Moz from crawling sites?
-
Hi, is it possible to stop Moz from crawling a site at the server level? Not that I am looking to do this or anything, but here's why I'm asking.
I have been crawling a site that is managed (currently by 2 parties), and I noticed that this week pages crawled went from 80 (last week) to 1 page!! I know, what? See my image attached... and the issues all went to zero "0"....!
So is it possible that someone can't prevent Moz from crawling the site at the server level? I checked the robots.txt file on the site, but nothing there. I'm curious.
-
Thank you both for the responses... I guess it was just a matter of time in my case. It looks like the keywords crawl/report comes in a few hours "after" the initial crawl email from Moz... not sure why, but about 15 minutes after I posted this question, I received the email titled "New Rankings and On-Page Reports Ready" that included the keyword rankings!
Thanks again
-
Here are the most common reasons why only one page is crawled. If this doesn't answer it, send us a note at help@moz.com. Thanks!
-
Ok, here is the helpful resource regarding Roger Mozbot. http://moz.com/help/pro/rogerbot-crawler
This might help!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I combat title tag issues in Moz because of long company/site name? [Currently using Yoast SEO Plugin]
I've only just started to dip my toe into the SEO pool, so please forgive me if this is a silly question. I've been working heavily in the Moz tool and the Yoast SEO plugin to clean up the meta descriptions on our site, as there were many pages without descriptions in place (we just transferred from a different CMS to Wordpress within the last month or so). Now I'm noticing on our Site Crawl that we have 60+ pages that have title elements that are considered too long. When I go in to fix these issues on our blog posts (as this is where most of the issues lie), Yoast is already giving me the green light on the title lengths. I've discovered that the reason why Moz is telling us that many of our site lengths are too long is because "| Parker Staffing Services" is added to the end of our title tags automatically. Deep breath Now that you've heard all of the backstory... Is there a way that I can prevent that site title (which is 26 characters all by itself! Talk about limiting!) from displaying on search engines when our blog posts or pages show up in search results? I've poked around in the Titles & Metas settings that come with Yoast, but I fear making too many changes will result in me screwing something up. Help an SEO beginner out? 🙂
Moz Pro | | giaciccone0 -
How do you get your web site recrawled with Moz without waiting for a week?
My initial crawl was screwed up because of a no follow that needed to be removed. I would like Moz to recrawl the site right away so I can find any other errors.
Moz Pro | | Ron_McCabe0 -
Crawl diagnostics taking too long
I started a crawl 2 days ago and it was still going after almost 48 hours so I deleted the entire campaign and resubmitted it. It's been 13 hours and still going. What happened to getting initial results in 2 hours? I've never had this problem and have run several campaign crawls here. Just wondering if there is a known issue I just can't seem to find? Thank you
Moz Pro | | LisaS130 -
Crawl Disgnosis only crawling 250 pages not 10,000
My crawl diagnosis has suddenly dropped from 10,000 pages to just 250. I've been tracking and working on an ecommerce website with 102,000 pages (www.heatingreplacementparts.co.uk) and the history for this was showing some great improvements. Suddenly the CD report today is showing only 250 pages! What has happened? Not only is this frustrating to work with as I was chipping away at the errors and warnings, but also my graphs for reporting to my client are now all screwed up. I have a pro plan and nothing has (or should have!) changed.
Moz Pro | | eseyo0 -
Usable to set up campaign because site cannot be
I don't understand this message. i never had problems with other sites and now I get problems with this message when trying to set a campaign twice for 2 different sites. I received the same message twice. What do I do? Help! We have detected that the root domain xxxxxxxxxxxxxxxxxxxx does not respond to web requests. Using this domain, we will be unable to crawl your site or present accurate SERP information. Thanks.
Moz Pro | | mcuneo0 -
Sites Blocking Open Site Explorer? Penguin related.
Last week I was looking at a competitors site who has a link scheme going on and I could actually check the links for each anchor text. This week they don't work at all, do you think they're blocking the rogerbot on their domains? Or is there a problem with open site explorer? http://www.opensiteexplorer.org/anchors?site=www.decks.ca If you're interested in the background, all the links are to instant-home-biz . com which then redirects decks . ca - it's a tricky technique. Pretty much all of the links are from sketchy sites like: airpr23.xelr8it.biz/ airpr23.anzaland.net/ airpr23.vacation-4-free.com/airpr23.blogfreeradio.net/airpr23.blogomatik.com/http://www.morcandirect.com/mortgages/resources2.php which I thought Penguin was supposed to catch…
Moz Pro | | BeTheBoss0 -
Set crawl frequency
Current crawl frequency is weekly, is it possible for me to set this frequency our-self?
Moz Pro | | bhanu22170 -
How do I get my crawl report?
I received a message that my crawl report is complete with a link - went to the link however - when I click on the icon that has the report name and the complete check mark nothing happens looked around can't find the results. Need to bid on this job so it would be helpful to know where to get it. Thanks for all you do. Wickey
Moz Pro | | Wickey0