Screaming Frog, Moz and other crawlers
-
Hi
Ignorant question, but is it possible to use Screaming Frog or the Moz crawler or any other reputable crawler for a site still in development i.e. it is yet to be indexed? If so, could someone provide some quick instructions on how this can be done.
Thanks in advance for any support.
Neil
-
Thanks for the answer, much appreciated.
-
Yes, thanks for the answer.
-
Tim,
As long as you can get to the site remotely via a website address you should be good to go. However, if the site is blocking crawlers via robots.txt file or meta robots tag rogerbot won't access it. On the other hand, screamingfrog has a setting to tell it to ignore the robots.txt file if one exists.
-
If the site is online or on a domain or somewhere where Screaming Frog can reach it you can spider it with the software yes.
But it has to be online somewhere so the software can reach it.
Did this answer your question?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I 'sign in' to the Moz Bar?
It's installed, I can see links etc as highlighted - but it won' t let me "sign in". This 20 second video explains: https://www.screencast.com/t/3kEjQFkTHZv Suggestions? Or shall I just ignore? Paul Barrs
Moz Bar | | PaulBarrs0 -
Moz Bar Truncated in Firefox
Hi, I'm using the Moz Bar add-on in Firefox.. but it's suddenly truncated meaning that I can only see half of the toolbar, and I'm unable to use it's features. Screenshot here; http://prntscr.com/bgm1z2 Does anyone know how to fix this? I've tried removing and re-adding Moz Bar, and installed a new version fo Firefox. Thanks in advance, Lee.
Moz Bar | | Webpresence1 -
Moz Page Analysis Country different to Who.is?
If I analyse a domain with Moz Page Analysis tool, it says that the domain is hosted in the United States but if look up the same domain on who.is, the hosting location is Italy?
Moz Bar | | Marketing_Today0 -
Moz Crawler not Identifying all Duplicate Pages
On two recent site crawls (9/27/14 and 11/4/14) for duplicate content the Moz tool did not ID the following 2 pages, which are 100% duplicate to each other: http://www.hooksandlattice.com/planter-hampton-241212.html ; Screenshot: http://screencast.com/t/DdwWroUU http://www.hooksandlattice.com/planter-hampton-721212.html ; Screenshot: http://screencast.com/t/8Lb1cJZmGrhX As I'm working feverishly to re-write and update the site (goal is ZERO duplicates) I'm finding it challenging to use the Moz tool to get the project done. Does anyone have any feedback or help they can provide for how I can identify all duplicate pages associated with my domain? Thank you! Lindsey Pfeiffer
Moz Bar | | CMC-SD0 -
I want to uninstall the Moz SEO toolbar. How do I do this?
I installed the Moz toolbar and I don't understand it and it covers up important parts of websites and makes them inaccessible. I want to get it off my computer. I installed it in chrome. How do I get it off?
Moz Bar | | Bonnie761 -
Moz Toolbar Make My Browser Slower Drastically!
Hi, I really like Moz toolbar but it's totally make my browser (Firefox 26) slower (open new tab, new site, close / open the browser, etc.) What can I do to make it works faster With Moz toolbar? Thanks
Moz Bar | | JonsonSwartz0 -
Does moz Ranking report canadian rankings?
Hi I would like to know Moz rankings can report canadian ranking or does it just focuses on US market search engines only? If it does also focuses on the canadian market, how do you set it up? Thanks
Moz Bar | | Ideas-Money-Art0 -
Can I delete a SEO campaign in Moz, and start a new one for a different website?
Would be nice to know, as I'm limited to 5 campaigns, and the most important work is done for 2 site's, so I would like to switch that to 2 other website's. Regards,
Moz Bar | | mrblue910