Screaming Frog, Moz and other crawlers
-
Hi
Ignorant question, but is it possible to use Screaming Frog or the Moz crawler or any other reputable crawler for a site still in development i.e. it is yet to be indexed? If so, could someone provide some quick instructions on how this can be done.
Thanks in advance for any support.
Neil
-
Thanks for the answer, much appreciated.
-
Yes, thanks for the answer.
-
Tim,
As long as you can get to the site remotely via a website address you should be good to go. However, if the site is blocking crawlers via robots.txt file or meta robots tag rogerbot won't access it. On the other hand, screamingfrog has a setting to tell it to ignore the robots.txt file if one exists.
-
If the site is online or on a domain or somewhere where Screaming Frog can reach it you can spider it with the software yes.
But it has to be online somewhere so the software can reach it.
Did this answer your question?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I use json-ld in our site but schema not found in markup tool bar moz
i use json-ld in our site. structure data in test without any error but in test markup moz in our site schema is not found! Is it time for Google to identify the code to verify the schema?
Moz Bar | | shokoufezand0 -
In Moz Campaigns, how are competitor domains tracked if they redirect their site?
Hello! One of our competitors (Company A) that we've tracked in Moz for a long time recently merged with another company (Company B) and redirected their whole site to Company B's site. Will our competitor tracking still work as-is? Or do we need to make an adjustment? I'm reluctant to delete Company A from our competitor tracking, because we will lose all of that data. But if all of the keywords are slowly going to drop off as Google starts showing Company B results only, it may be the only option. Any help is appreciated! Thanks!
Moz Bar | | PrimeFoodTeam0 -
Moz is only crawling 2 pages
Hi, I found a similar thread, but it did not provide a clear-cut answer. We have had this campaign running for over a year, and we are always adding content to the website, but Moz is only ever able to crawl 2 pages, Screaming Frog only picks up 12, but I know there is a lot more than that. None of our pages are set to no-index, so I do not know what is causing this. Welcoming any ideas/solutions. Thanks
Moz Bar | | GavinAdv0 -
GA snippets for subdomains and best tracking in MOZ
Hey there, I have a blog running at blog.URL.com. I need to add Google Analytics and I want to track the value of this subdomain as separate from the main URL. Can I do that? If I do a unique snippet in the header of the blog is that the right way to go? The main site is on an antiquated CMS system and will be converted over the next few months to a full on WP site, does that change anything long-term? Thanks in advance for your help. Cheers, Lisa
Moz Bar | | LisaBOS0 -
Moz Crawl Showing Duplicate Content But It's Not?!
Unfortunately I can't give out the URL, but here's the deal... I have two URL's which have completely different content on them but are being crawled as duplicate content. Any Idea how that would happen? I'm not seeing any errors in WMT's. Has anyone seen this before? Is the duplicate content reporting based on a % of the page content matching as the same?
Moz Bar | | Swarm-SEO0 -
Rel Canonical and Moz Crawl
we have Rel Canonical tags set up on a few pages. When viewing the page source, the tags are correct. However, Moz Crawl results show the opposite. for example the page source, correctly shows, URL X with a Rel canonical Tag of URL Y
Moz Bar | | S.S.N
but.. Moz crawl is showing URL Y with a Rel Canonical Tag of URL X ..any thoughts why this would happen? which should i trust more?0 -
Why isn't seo moz properly crawling my site?
In my campaign results, only 2 pages have been crawled, when there are many more. Also, when I do the on-page evaluation for my homepage and the main keyword, I get a grade of F, with a ton of errors that my page actually does not have. It's as if SEO moz is having trouble crawling through my site. Any ideas on this?
Moz Bar | | diplomajim0 -
What happened to moz Crawl Test? Is it moved in the redesign?
Love the new design of moz.com! And the mentions tool is terrific. However, I use the Crawl test a ton and cannot find where to access it now.
Moz Bar | | jessential0