How do get Moz to spider a Development site PRE LAUNCH?
-
Hi,
Does anyone know how we could get Moz to browse a development site before launch? But without Google and other engines indexing it?
Thanks
-
That would do it! Try this in robots.txt:
User-agent: Rogerbot
Disallow:User-agent: *
Disallow: /Keep in mind that this will only block crawlers that obey robots.txt. Once Roger crawls your site, it may be a good idea to lock the dev site down entirely.
-
Well, you could block Google and Bing from crawling your website (via Robots.txt) and run a Crawl via Moz, that's possible I think.
Istvan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
KD according to MOZ.
Hello, I want to know how moz calculates the keyword difficulty? What are the basic things which moz pro consider while calculating keyword difficulty? I wanna Calculate the KD of JB Battery Keywords,
Getting Started | | Shafqat_mehmood0 -
Can Moz Monitor a JS Site?
We are building a new site that, on the blog landing page, uses JS to populate the individual blog article links on the page. The links are not viewable in the page source, but do appear once the page fully renders. After running an on-demand crawl of the site (in QA...not indexed yet), it appears that Moz isn't indexing these pages, and it also isn't reading other page elements that load later (like an H1 that is rendered in JS but not in page source). Are we going to be able to use Moz to track this site? Is there some setting to help?
Getting Started | | Rodrigo-DC0 -
Why does the moz bar link count differ from the moz link analysis page
Hi all, Why does the Moz Bar show a different link count from the Moz link analysis page? For example, when I check the SERP below, for the first result, the bar shows 936 page links from 4 RDs. But when I check out the link analysis page, it tells me there are just 141 page links from 4 RDs. What gives? For the second entry, the bar shows 6 page links from 0 RDs. Not sure how that's possible. Can anyone explain these things. Thanks! Andy Reviewed SERP: https://www.google.com/search?source=hp&ei=hhRVW5yyH5C60PEP-_-isAw&q=mountain+bike+trails+near+me&oq=mountain+bike+trails+near+me&gs_l=psy-ab.3..0l7j0i22i30k1l3.645.4654.0.5322.28.19.0.8.8.0.243.2920.3j11j5.19.0....0...1c.1.64.psy-ab..2.26.2961...0i131k1.0.C4lAxLkGgH0
Getting Started | | AndyKubrin0 -
I haven't used Moz and you've been billing me. Please refund thanks
I signed on for a free trial and found out today that you've billed me in June and July. Check my history as this is only the second time i've logged in and it's just to request you refund.
Getting Started | | Completebanker0 -
Moz for charities and non-profits
Hi, I noticed an earlier thread on this topic that now seems dead so I decided to start a new one: https://moz.com/community/q/non-profit-rates We're a non-profit organisation based outside of the US and we were wondering if MOZ's non-profit program can be extended out? We're fully registered and audited in our home country and can provide all the necessary documentary proof if needed.
Getting Started | | OBW_Noel1 -
Why I can't add more campaign to my moz analytics?
As we know that being a standard pro member in Moz community, we have at least 5 campaigns to add and track & as well in new moz analytics. But when I just made http://www.norwoodgreen.in website campaign an archive to make active or add one of the old himachalpackagetour.com campaign which I was unable to do WHY WHY WHY? Let me know . .? Also do guide or train me How to use Moz analytics as I was not enough familiar with it as I am habitual with in Pro tools. Best,
Getting Started | | Futura
Teginder0 -
Moz Analytics set up: should I use www or not when I do website set up?
Hello, When doing website set up will www.mysite.com yield different results than just mysite.com ? Same about competitors: should I use www or not when I set up their sites? Or it won't matter? Thanks,
Getting Started | | dstepans
Denis0