Screaming Frog, Moz and other crawlers
-
Hi
Ignorant question, but is it possible to use Screaming Frog or the Moz crawler or any other reputable crawler for a site still in development i.e. it is yet to be indexed? If so, could someone provide some quick instructions on how this can be done.
Thanks in advance for any support.
Neil
-
Thanks for the answer, much appreciated.
-
Yes, thanks for the answer.
-
Tim,
As long as you can get to the site remotely via a website address you should be good to go. However, if the site is blocking crawlers via robots.txt file or meta robots tag rogerbot won't access it. On the other hand, screamingfrog has a setting to tell it to ignore the robots.txt file if one exists.
-
If the site is online or on a domain or somewhere where Screaming Frog can reach it you can spider it with the software yes.
But it has to be online somewhere so the software can reach it.
Did this answer your question?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Am I the only one seeing the Moz's points ranking with mixed rankings?
Hello everyone. It's been several weeks since the problem persists, Here: https://moz.com/community/users
Moz Bar | | Gaston Riera
There are rankings that are incorrect and some users that have the same ranking than other users. I've attached some pics. GR 25603b49292acd5d68f7dd6a43977f94 6d8fea557f6ed2de23466f6c7df850eb0 -
How to Find Related Keywords in Moz.com
Hello, I have used the SEMRush tool and it provides related keywords with other metrics. How do I find the same using Moz.com.
Moz Bar | | Chaku0 -
Link to hotels on http://moz.com/mozcon doesn't work
Hi The link to the hotel for Mozcon 2015 doesn't work - seems like its the 2014 link still in place. Thanks Andy
Moz Bar | | Andy-Halliday0 -
Does anyone else have issues with Moz's keyword search volume tool for Google's search engine?
It will show the search volume for Bing even when Google is selected. Then, if you select Bing, you'll get the same data as it shows for when you selected "google". So basically, this tool does not work for Google's search engine. Or it is most likely not a reliable way to perform keyword research. Anyone else notice this? Does Moz even offer a way to submit a support ticket to get this fixed?
Moz Bar | | ShokIdeaGroup1 -
How do you feel when Moz marks one of your questions as "answered?"
Hi everyone, This is not meant to be snarky at all, so I just want to preface my question with that. So, since the new re-branded Moz rolled out last year, I'm sure many of you have noticed that if you ask a question and it is answered by a Moz associate, your question is marked as "answered." I'm sorry, but I don't like this. Here's why, I'm the one who asked the question. I should be the one who determines if the answer was adequate for me, or if it didn't sufficiently answer my question. This is particularly true when my question doesn't have to do with a customer service issue or a Moz tool question. If I ask a question about SEO, Content, CRO, marketing or any other subject, I feel like it should be me and only me who determines whether or not I feel like my question is answered. In addition to this, Moz is actually depriving themselves of useful UGC by shutting down questions in this way. How? Because when the rest of us who frequent the Q & A see a question that's already been marked as "answered" we tend not to open it, read it and respond, because we think that person has already gotten what they needed....when in fact, it could be that a Moz associate has jumped in and marked their question as answered when it really wasn't. Consequently, we all miss out. I propose/move that Moz associates can only mark questions as "answered" when they pertain directly to Q & A about Moz tools, service and support. All other questions must be marked as "answered" only by the asker or closed as "answered" after they have been dormant for 6 months or more. Can I get a second (motion) ?
Moz Bar | | danatanseo4 -
Beta Moz is losing my campaigns.
I've been trying to make the transition to Beta especially now that Pro has such serious problems with the style sheet. The thing is, all of the campaigns I have added to the Pro side don't show up in the Beta side. They'll show in Pro but that's all but unusable on my end and there's no Archive button visible on Beta. I tried adding a campaign again to Beta thinking it just isn't there, but it tells me it's already been added. Well then were is it? I just added them earlier today! I can see them on Pro but nowhere on Beta, nor the other 3 I have added on Pro in the last 2 weeks.
Moz Bar | | seowhat0 -
Moz keywords tool obsolete?
It looks like Google is going to encrypt all user searches, rendering entire sections of SEO tools useless like portions of Moz. What's Moz's reaction to something like this? http://blog.hubspot.com/google-encrypting-all-searches-nj
Moz Bar | | BlueLinkERP0 -
When Moz provides an on-page grade report card, what tells Moz which keyword to grade for?
I just started working with the on-page grader and of couse am particularly interested in fixing my pages with not so stellar grades. Can someone tell me how SEO Moz decides what keyword to grade a page for? I receive a report each week that tells me how many pages I have with A's, B's etc. But what tells Moz to grade a page for a particular keyword? Is it the first keyword listed in the meta description?
Moz Bar | | AliciaMarie0