Screaming Frog, Moz and other crawlers
-
Hi
Ignorant question, but is it possible to use Screaming Frog or the Moz crawler or any other reputable crawler for a site still in development i.e. it is yet to be indexed? If so, could someone provide some quick instructions on how this can be done.
Thanks in advance for any support.
Neil
-
Thanks for the answer, much appreciated.
-
Yes, thanks for the answer.
-
Tim,
As long as you can get to the site remotely via a website address you should be good to go. However, if the site is blocking crawlers via robots.txt file or meta robots tag rogerbot won't access it. On the other hand, screamingfrog has a setting to tell it to ignore the robots.txt file if one exists.
-
If the site is online or on a domain or somewhere where Screaming Frog can reach it you can spider it with the software yes.
But it has to be online somewhere so the software can reach it.
Did this answer your question?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved New Moz Bar not working in my domain
Hello, I have installed the new version of Moz Bar in Google Chrome.
Moz Bar | | Alberto D.
My problem is that it analyzes all domains except mine: https://elchesemueve.com
I also can't find my country (Spain) to configure the site.0 -
Incorrect country flag renders in Moz Bar. Why and how do I remove it?
The Moz Bar displays a german flag un front of the URL in On-Page elements and lists Germany as the Country location in General attributes. Why is this, and how do i change it?
Moz Bar | | eocreative0 -
Moz Rankings vs DuckDuckGo
Hello, So we have a client that’s only interested in a handful of keywords. We would like track his rankings through a Moz Campaign but he currently uses Duck Duck Go to check them. Strictly from an accuracy perspective (and not necessarily added features like Mobile Rankings) is there any arguments we could use to convert them to Moz? If not we will let them know but any input would be appreciated. Looking forward to hearing from all of you! Thank you in advance for the help. Best,
Moz Bar | | Ben-R0 -
Perplexed by last MOZ crawling duplicate content errors
In the last crawler issues report from MOZ I can see many many pages listed as duplicate content with 0 duplicate urls. Like this: http://imgur.com/fbikRVq I am puzzled, what does it mean?
Moz Bar | | max.favilli0 -
Can you give some examples of MOZ Local Partners who have an API and update quickly?
In answering questions about the speed of MOZ Local, I've seen this a few times: "The time it takes our partners to update the listing really depends on whether or not they have an API. If they do have an API, the listing will be updated pretty much immediately." Can someone provide a list of which Partners have an API, and which do not?
Moz Bar | | irapasternack0 -
Weird back link showed in moz crawl
Some time ago somebody from this site: http://dianibeach.com created a weird link to our site which had on the end db. Later we have realized that the link was coming from every footer on each page. I believe that the back links from footer does not have realy value and even the more of them the less value. We have asked the guy to remove that links as I thought it might harm our site more then help. Now I I was very surprised to find this link in moz crawl error as second top page on our site in current index??? Can somebody explain how is this possible?? The most ridiculous thing is that when I click on that link it realy opens our site! How is that possible, what is it? This is the link: http://villasdiani.com/?db Thank you very much for any help with this
Moz Bar | | Rebeca10 -
Does Moz Analytics Allow for Goal Tracking?
We're currently tracking goals in GA, but would be interested to learn if that can be accomplished in Moz Analytics.
Moz Bar | | CGalownia0 -
Moz.com Report a Problem or Provide Feedback
Two things. First, I could not find exactly where to report an issue or provide feedback about the website. I don't need to contact about Moz Analytics or any other products. I just want to provide feedback about the Moz Blog. I think it would be nice to be able to report an issue or provide feedback to "Roger" and have that item sent to the appropriate department for review. A lot of websites tend to use "webmaster@example.com" for this, but anything would work. My suggestion is to offer a feedback form on the Contact Moz page. Second, the issue I wanted to report has to do with blog comments. I left a comment early this morning and, with me being on Central Time and the timestamp for comments using Seattle local time, I was surprised to see that my comment had been posted "2 hours ago" immediately after I clicked "Submit." I presume this text is dynamic and that, in Seattle, you see "seconds ago" or something but, being in Wisconsin, I see "2 hours ago." This is confusing mainly because I have no idea how long ago other comments may have been posted. P.S. There isn't a question category for the Moz Blogs so I am choosing Moz - Other Research Tools. I view the blogs as a Product Moz provides so maybe adding "Moz - Blogs" to the Moz Product Help list of categories would be appropriate.
Moz Bar | | GeorgeAndrews0