Screaming Frog, Moz and other crawlers
-
Hi
Ignorant question, but is it possible to use Screaming Frog or the Moz crawler or any other reputable crawler for a site still in development i.e. it is yet to be indexed? If so, could someone provide some quick instructions on how this can be done.
Thanks in advance for any support.
Neil
-
Thanks for the answer, much appreciated.
-
Yes, thanks for the answer.
-
Tim,
As long as you can get to the site remotely via a website address you should be good to go. However, if the site is blocking crawlers via robots.txt file or meta robots tag rogerbot won't access it. On the other hand, screamingfrog has a setting to tell it to ignore the robots.txt file if one exists.
-
If the site is online or on a domain or somewhere where Screaming Frog can reach it you can spider it with the software yes.
But it has to be online somewhere so the software can reach it.
Did this answer your question?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz DA is not increasing
Hey my website best offset smokers under $1000 is not increasing DA. is there anyway to increase the DA currently its DA is 8
Moz Bar | | 0hjgh60 -
Moz Crawl only crawling the top level page (1 page)
For the past few mounts my weekly site crawl has been inconsistent. One week works fine, it crawls all of my 500 or so pages. The following week it only crawls 1 page (http://mydomain.com) and nothing else. A few weekly scan go by and the crawl is back up the the 500 or so pages.I went ahead and created several campaigns with duplicate settings and crawled the site. Most times but not all the new campaign's crawl works fine crawling all pages. But within a week or two the weekly crawl will fail again. (crawling 1 page). Currently i have four campaign's all with the same settings running weekly crawls. 2 campaign's crawled the 500 pages and two crawled only the single page. Any help will be greatly appreciated
Moz Bar | | dmaude0 -
Incorrect country flag renders in Moz Bar. Why and how do I remove it?
The Moz Bar displays a german flag un front of the URL in On-Page elements and lists Germany as the Country location in General attributes. Why is this, and how do i change it?
Moz Bar | | eocreative0 -
All in favor of an "Announcements" category section say MOZ!
So this is just a thought. I am not Rand, therefor I am not the Moz man himself, and therefor I am not telling anybody how to run this website or forum. But, I've been thinking... and usually that a dangerous thing. So many times when I answer a question I end it with: "Keep me posted" "Keep us updated" "Good Luck" In the same light there have been plenty of times when I have had my questions answered by people like EGOL, RobertFIsher, KeriMorget and they have said that they have wanted me to keep them posted. Now sure, some of you are sitting there being Melancholy Mozzers thinking that all those wonderful people are just being polite by adding some nice parting words... but what if I they are being Sincere Seoers? What if I WANT to fill them in and don't want to ask a question? What if... THEY REALLY LIKE ME?!?! I could also send a DM to those 3 people, I mean let's get honest I'm not THAT popular.... but I can dream can't I? Again, this isn't really an SEO question, more of a thought on where I'm supposed to give updates on my project. In all honesty I do want feedback on what I'm doing, and sometimes I don't even know what questions I'm supposed to ask!
Moz Bar | | HashtagHustler1 -
Problems with Moz tools
Hello, I am having real problems with Moz tools. I can log in to Moz analytics (crawl diagnostics) and analyse one site and I get 2 [internal] duplicate page errors Yet when I enter via pro.moz.com (crawl diagnostics) I am finding over 100 duplicate page content warnings? Any idea why this is happening?
Moz Bar | | McTaggart0 -
Beta Moz is losing my campaigns.
I've been trying to make the transition to Beta especially now that Pro has such serious problems with the style sheet. The thing is, all of the campaigns I have added to the Pro side don't show up in the Beta side. They'll show in Pro but that's all but unusable on my end and there's no Archive button visible on Beta. I tried adding a campaign again to Beta thinking it just isn't there, but it tells me it's already been added. Well then were is it? I just added them earlier today! I can see them on Pro but nowhere on Beta, nor the other 3 I have added on Pro in the last 2 weeks.
Moz Bar | | seowhat0 -
Moz Analytics Beta
Just got the access to this, but my campaigns don't seem to be showing. Anyone know why this might be?
Moz Bar | | Jonathan19791 -
Can the Moz tool identify variations of a Chinese language branded keyword?
We've recently started a trial of the Moz Pro service and are tracking a selection of keywords. Our primary band / product name is 功夫英语(Kungfu English), so we've set a rule that any keywords containing those four Chinese characters (功夫英语) should be marked as a branded keyword. However, in the "Non-Paid Keywords Sending Search Visits" section of our traffic report, we see a few variations on our brand name that are not being marked as branded keywords. (See attached Images). Based on our rules, shouldn't these variations also be marked as branded keywords without our needing to manually add them as such? Or have I misunderstood the intent of this rule? For the English text, the brand rule about words containing "kungfuenglish" seems to have resulted in all of "www.kungfuenglish.com", "http://www.kungfuenglish.com", and kungfuenglish being labelled as branded keywords. However, I'm not seeing the same sort of result with variations on the Chinese keyword, 功夫英语. KedTi1D.png z5bcUIt.png
Moz Bar | | PaulCoffey0