Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

How do get Moz to spider a Development site PRE LAUNCH?

Getting Started

735

bjs2010 last edited by

Hi,

Does anyone know how we could get Moz to browse a development site before launch? But without Google and other engines indexing it?

Thanks
1 Reply Last reply
Reply Quote 0
MattRoney @Keszi last edited by

That would do it! Try this in robots.txt:

User-agent: Rogerbot
Disallow:

User-agent: *
Disallow: /

Keep in mind that this will only block crawlers that obey robots.txt. Once Roger crawls your site, it may be a good idea to lock the dev site down entirely.
1 Reply Last reply
Reply Quote 1
Keszi last edited by

Well, you could block Google and Bing from crawling your website (via Robots.txt) and run a Crawl via Moz, that's possible I think.

Istvan
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

I want to increase the DA of my Site

Dear all i have established a new site and want to increase its DA fast. So i need your suggestions in this regard. This newly established blog is my first independant project so i want to make it perfect.
Getting Started | | hamza5252

0
Moz site crawl doesn't work

The Moz site crawl isn't working for my campaign, but works for the site's on demand crawl. The search should not be disallowed by robots.txt or the headers. I'd like to be able to track the website for the campaign so I can see SEO gains / losses and increases / decreases in indexing.
Getting Started | | DrainKing

0
Does MOZ pick up every issue in one crawl?

Hi, Does MOZ pick up every error/warning in one crawl? Or does it take numerous crawls? Many thanks Lee
Getting Started | | lbagley

0
Getting Started: Multiple Campaigns VS Single Campaign

I'm new to Moz and am trying to determine whether to create multiple campaigns broken out into product categories or a single campaign and label keywords by product category. I figure the majority of the folks in the Moz Community are in agency space, so most probably prefer a single campaign per client. We're an in-house marketing team so we have the ability to track up to 10 campaigns but only need to track a single website. I like the idea of having the different campaigns for different product categories, but I don't want to make things overly complex either. Just because you can do something doesn't necessarily mean you should.
Getting Started | | DFrey

1
Can't track my site, keep getting "Ooops. Our crawlers are unable to access that URL"

Hello, So i keep getting this message and I went to hurl.it and I get 200 response. But it appears its not my actual homepage bc it says the body is empty and in the title it says "COMING SOON" which is not what my actual homepage says. Does anyone know what this means?? Thank you in advance! Rena
Getting Started | | Palila-Studio

0
How long will it take to get google Analytics results back so I can get rolling?

I just signed up for Moz. I'm wondering how long it will take until Google Analytics results will enter.
Getting Started | | John-Francis

0
Need to Add Moz code?

Do i need to add tracking code in my website, for Moz tool to analyze my website. Please someone let me know.
Getting Started | | TOBOC

0
Google Analytics data not loading into Moz

I use several analytics apps on my iphone which require you to connect to your GA account. Each one gets immediate access to the GA data. With Moz, I connected to my GA account 3 days ago after many failed attempts due to some bug, but still have no data showing. Is there some reason why Moz would take several days to access the data where iphone apps take seconds? I am now a week into my 30 day free trial, and have spent lots of time trying to get it to work, but have not been able to do much. I'm afraid of how much of the 30 days will be taken up just getting Moz set up and working, rather than actually using and evaluating what it does. Any possibility of re-starting the 30 days from when it's actually working? I don't mind waiting patiently for some bugs to be fixed, just would like to know that I will still get a 30 day trial. Thanks.
Getting Started | | RLodato

1