Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

How do get Moz to spider a Development site PRE LAUNCH?

Getting Started

784

bjs2010 last edited by

Hi,

Does anyone know how we could get Moz to browse a development site before launch? But without Google and other engines indexing it?

Thanks
1 Reply Last reply
Reply Quote 0
MattRoney @Keszi last edited by

That would do it! Try this in robots.txt:

User-agent: Rogerbot
Disallow:

User-agent: *
Disallow: /

Keep in mind that this will only block crawlers that obey robots.txt. Once Roger crawls your site, it may be a good idea to lock the dev site down entirely.
1 Reply Last reply
Reply Quote 1
Keszi last edited by

Well, you could block Google and Bing from crawling your website (via Robots.txt) and run a Crawl via Moz, that's possible I think.

Istvan
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Unsolved What are the top 5 things you recommend you should do when you first subscribe to Moz?

getting started

Hi
Im new to Moz and would like to get the most out of it as quickly as possible. so like the subject say - What are the top 5 things you would recommend I should to get the most out of Moz? #getstarted
Getting Started | | djnlowe

0
I used moz tool and it was very useful to me.

I used moz tool for 3 months. Before that my website never ranked. But now it started ranking. Thanks to moz. My website is https://www.designcafe.com
Getting Started | | designcafe

1
Moz only crawling one page of a campaign, please help

Today I set up a new campaign for a client, however the crawl has only found the home page and is saying that the URL is unavailable. The site is definitely live and the URL is correct. I have set up the campaign 3 times one with the full address (http://www.) one with www. and with just the domain name. All three of these have come page with one page crawled and "unavailable" above the URL. It is picking up the crawl issues on the page and showing domain authority but I don't know why it's not crawling other pages. Prior to setting up the campaign I did a site crawl and Moz found everything then, so I don't know why it isn't now. Please help. Thanks
Getting Started | | Wrapped

0
Getting Started: Multiple Campaigns VS Single Campaign

I'm new to Moz and am trying to determine whether to create multiple campaigns broken out into product categories or a single campaign and label keywords by product category. I figure the majority of the folks in the Moz Community are in agency space, so most probably prefer a single campaign per client. We're an in-house marketing team so we have the ability to track up to 10 campaigns but only need to track a single website. I like the idea of having the different campaigns for different product categories, but I don't want to make things overly complex either. Just because you can do something doesn't necessarily mean you should.
Getting Started | | DFrey

1
Moz Not Pulling Monthly Data

So I set up my Moz campaigns on the 8th of this month. I can see data for the week, however, I can't see data from the previous week or for the month. So my question has two parts: a.) Is there a way to view data for the previous week, and b.) when can I expect to see data for the entire month. Seeing data for the entire month is absolutely critical for me, since this is what I'm going to use to prepare my monthly client reports. Has anyone experienced anything like this? PLEASE HELP!!!
Getting Started | | maxcarnage

0
Does Moz provide more than just the SEO Tool?

We've received a lot of good info about our website using the Moz analytic tools but I'm curious to know if Moz provides consulting services on how to implement changes based on the data. Basically, Moz gave me the "what", I need some direction on the "how" and does Moz have the in house capabilities to assist with the "who" can do the actual work.
Getting Started | | Elara

1
Can I upgrade my moz subscription from large to premium without getting my campaigns being deleted?Do i need to wait till the end of the month to resubscribe?

I'm planning to take a large subscription of moz..but i may need a premium account in the future.Can I upgrade it in the middle of my current subscription or will i have to wait till my subscription gets over? Also, I'm planning to take only one month currently but i'll renew it once the month completes so can i continue the same account without my campaigns being deleted?
Getting Started | | kdcdmp

0
How to get moz to crawl a staging domain that is blocked by robots.txt

Is it possible to get Moz to do a crawl report on a domain blocked by robots.txt and actually display all errors instead of only one saying the domain was blocket in robots.txt? Anything i can add to robots.txt to make moz able to do the crawl report but still hinder google from crawling a staging domain?
Getting Started | | classifiedtech

0